spec: define initialization order more precisely

[the-zucc]

Context

Package Initialization

When importing a package, it is initialized according to the sequence described in the Go specification. If that package has dependencies, it makes sense that these are initialized first, and that is what is described in the spec. In many instances, this is very useful and helps to write very succinct code, eliminating much boilerplate code.

Importing for Side Effects

In the Go specification, there is mention that it is possible to import packages for their side-effects:

An import declaration declares a dependency relation between the importing and imported package. It is illegal for a package to import itself, directly or indirectly, or to directly import a package without referring to any of its exported identifiers. To import a package solely for its side-effects (initialization), use the blank identifier as explicit package name:

import _ "lib/math"

Small Issue

Currently, the import order is predictable for packages with import relations. This is very useful and works flawlessly. However, as I've come to realize it, within a series of import statements in a single file, a package that is imported (for its side effects) early in the list of imports has no guarantee of early initialization. Consequently, the initializations are not ran in order, which makes results unpredictable at times.

Example Use Case - Mocking An Execution Environment Before Running Unit Tests

The package names here are not necessarily representative of what has caused this issue, they are just to explain the use case; let's say I have a package, packagea, that initializes a connection to a server, either in its init() or via variable initializers.

package somepackage_test

import _ "github.com/the-zucc/packageb" // this init() function sets an environment variable containing the server URL
import "github.com/the-zucc/packagea" // this package's init() function reads the environment variable

// Just imagine some unit tests here... :)

In this case, I've observed 2 things:

• There's no guarantee that packageb will be imported and initialized before packagea -- in my case, the package imported the earliest was initialized at the end, effectively not propagating its side effects early enough in the program execution for them to apply to other package initializations, as the environment variable was not set which caused packagea to default to the "production code" server URL, and not the testing URL;
• There isn't anything in the documentation or in the official Go specification referring to package initialization order within a series of import statements, or within a grouped import.

Proposed Solution

Enable some way of specifying that a package should be imported before any other package for its side effects. As an example, initialization behavior for grouped imports could remain the same, but for separate import statements, statement order could define the initialization order.

import _ "github.com/the-zucc/packageb" // this could be initialized first

import ( // and then the initialization behaviour could remain the same for this grouped import
    "github.com/the-zucc/packagec"
    "github.com/the-zucc/packagea"
)

EDIT: as mentioned in the comments, the proposed solution wouldn't work, unfortunately; however it still gives an idea of what a candidate implementation could look like.

[ianlancetaylor]

CC @griesemer

[seankhliao]

Given the widespread use of goimports and similar tools, the proposed solution won't work as it would be rewritten into a single block with sorted ordering.

More generally, what if a package does:

import _ "github.com/the-zucc/packageb" 

import (
    "github.com/the-zucc/packagec"
    "github.com/the-zucc/packagea"
)

while another does

import _ "github.com/the-zucc/packagec"

import (
    "github.com/the-zucc/packageb"
    "github.com/the-zucc/packagea"
)

or even different files in the same package?

[seankhliao]

see also discussions in the other direction, pointing to perhaps the more common need to protect against people writing init functions

removing init #43731
control over defering init #48174
import without running init #38450

[the-zucc]

Given the widespread use of goimports and similar tools, the proposed solution won't work as it would be rewritten into a single block with sorted ordering.

Depending on the implementation of this, goimports could need an update anyway.

More generally, what if a package does:

import _ "github.com/the-zucc/packageb" 

import (
    "github.com/the-zucc/packagec"
    "github.com/the-zucc/packagea"
)

while another does

import _ "github.com/the-zucc/packagec"

import (
    "github.com/the-zucc/packageb"
    "github.com/the-zucc/packagea"
)

or even different files in the same package?

That is a great question... couldn't this raise a syntax error ? Like a "pick a lane bud, you're drunk !" 😄

[seankhliao]

packaged does the first one, packagee does the second
you import both packaged and packagee, but not any of packagea, packageb, packagec directly

are you just never allowed to import these two packages together?

[the-zucc]

packaged does the first one, packagee does the second you import both packaged and packagee, but not any of packagea, packageb, packagec directly

are you just never allowed to import these two packages together?

Without having thought about it too much (therefore don't take my word for it), I think I would simply raise a syntax error on both import lines (for packaged and packagee...) There is hardly any reason why one would want to have 2 different import orders, and if there was a reason, that reason doesn't exist as of today, because at the moment, package initialization order isn't guaranteed (and so we can eliminate that problem by raising a syntax error.)

[the-zucc]

There is hardly any reason why one would want to have 2 different import orders, and if there was a reason, that reason doesn't exist as of today, because at the moment, package initialization order isn't guaranteed (and so we can eliminate that problem by raising a syntax error.)

Actually, I see a potential problem with applications with huge dependency trees with open source packages. One would want to take this into consideration if something is to be done...

[rittneje]

You should not connect to a server from init() as that violates the concept of writing testable code, as you have encountered.

Instead, packagea should have a proper constructor that gets invoked. Each unit test is then responsible for calling it. This establishes proper isolation between each test case, which is a very useful property.

Meanwhile, unit tests for other packages that depend on packagea should likely be using some interface instead of direct function calls, which allows for mocking.

[griesemer]

See also #31636. You can force the init order by ordering the imports accordingly and into "blocks" so that they don't get re-ordered (example). Note that this is not prescribed by the language, it's an implementation behavior.

That said, it is much more robust to be explicit and call initialization functions explicitly if they cause side effects.

[the-zucc]

You should not connect to a server from init() as that violates the concept of writing testable code, as you have encountered.

I guess you are right, there are probably other ways to get succint code without connecting to a server in the init() function. However the fundamental question regarding side effects is still of interest, for other use cases.

[seankhliao]

it would be useful to have real usecases motivating these discussions.

[rittneje]

However the fundamental question regarding side effects is still of interest, for other use cases.

This proposal is only useful if package A's initialization is dependent on some other package B, despite the fact that A does not import B directly or indirectly. Writing code this way seems extremely confusing and error prone, especially because it will silently break if B imports A directly or indirectly, since then A would have to be initialized first no matter what.

[the-zucc]

See also #31636. You can force the init order by ordering the imports accordingly and into "blocks" so that they don't get re-ordered (example). Note that this is not prescribed by the language, it's an implementation behavior.

That said, it is much more robust to be explicit and call initialization functions explicitly if they cause side effects.

I'm glad that in the example you provided, the same assumptions are made; however while testing this I experienced some irregularities.

• Ordering imports as you have shown in the example does not seem to work for test packages;
• The widespread goimports tool breaks intentional import rearrangements; if I force an import order, and run the code before goimports rearranges it, it works (for a main function at least.) But if I run goimports (as is done automatically in VS Code,) it rearranges it and it breaks the intended order. For that specific issue I'll probably raise an issue in goimports if it turns out to really be an issue

[griesemer]

VSCode's goimport doesn't rearrange imports in different groups if they are separated by a blank line:

import (
	_ "b"
	// blank line
	_ "a"
)

(at least for me.)

That said, let me emphasize again: it's much better to explicitly call initialization functions that have important side effects if you want robust code.

[the-zucc]

VSCode's goimport doesn't rearrange imports in different groups if they are separated by a blank line:

import (
	_ "b"
	// blank line
	_ "a"
)

(at least for me.)

Thanks for chiming in; as I've mentioned, this doesn't seem to affect import order consistently for unit tests. It initializes packages in the proper order during normal execution, but when running unit tests, this order is not followed;

Let me share an example:

-- go.mod --
module somepackage

go 1.19
-- somepackage.go --
package somepackage

import (
	"os"
)

var ENV_VAR = os.Getenv("ENV_VAR")
-- mock/mock.go --
package mock

import "os"

const ENV_VAR_VAL = "localhost"

var _ = os.Setenv("ENV_VAR", ENV_VAR_VAL)
-- somepackage_test.go --
package somepackage_test

import (
	"testing"

	"somepackage/mock"
	_ "somepackage/mock"

	"somepackage"
)

func TestSomePackage(t *testing.T) {
	if somepackage.ENV_VAR != mock.ENV_VAR_VAL {
		t.Fail()
	}
}

If I run this code on my machine (i.e. run the unit test TestSomePackage(),) it shows that Go does not initialize somepackage/mock before somepackage.

[the-zucc]

I also just confirmed that in the example above, running the tests in a separate test package causes the behavior to change:

// test/test_test.go
package test_test

import (
	"testing"

	"somepackage/mock"
	_ "somepackage/mock"

	"somepackage"
)

func TestTests(t *testing.T) {
	if somepackage.ENV_VAR != mock.ENV_VAR_VAL {
		t.Fail()
	}
}

In this case, the desired import order is the same, but the result is not the same, and the test passes.

[griesemer]

What matters here is also in which order the files are presented to the compiler by the go command.

But I cannot but emphasize one more time: depending on these orders leads to extremely fragile code. If there are important side-effects that your code depends on, they should be encoded explicitly by calling respective initialization functions in the desired order.

[the-zucc]

That is true, and I agree; the practice should be avoided in most instances. However, the problem of reproducibility and consistency still remains. The commands I'm using to run unit tests is go test somepackage and go test somepackage/test, and so I'm not passing any files to the compiler "manually".

• As it stands, simply moving a unit test file from the module's root package to a subpackage changes the import order. Should it be that way ?
• According to the Go specification, packages can be imported for their initialization's side effects. However, these side effects are not guaranteed to run in a specific order during initialization, because this isn't part of the spec and is implementation-specific. In that case, wouldn't it be more coherent to simply discourage the use of that functionality in the Go spec ? or remove it from Go altogether, and recommend that initialization functions are called explicitly, like you are saying here ?

I'm just trying to understand why it was, that with all my careful research and attentive reading of the Go spec, I ended up believing that I can have some side effects propagated during the import process; and then ended up in the end realizing that the import order is inconsistent, and that the import for side effects functionality doesn't guarantee any import order and so should be avoided altogether.

If I am to import some package for its side effects, but can't have anything from the import/initialization process depend on it (implicitly), well then isn't the import for side effects functionality rendered useless, since we can't derive any assumption from it? I mean that, if the only code that can rely on side effects from imports is the one that sits in the package that is importing for side effects, couldn't the imported package (the one with the side effects) simply expose a method that is called in the init process of the importing package ?

I'm just thinking that if calling the initialization manually in my package is a better approach, then importing a package for its side effects isn't really all that useful and should probably be discouraged; am I missing something ?

Should a mention about this possible inconsistency be included in the spec, in the section where import for side effects is brought up?

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[ianlancetaylor]

If we do anything here I think we should do the algorithm discussed in #31636. That issue was closed with a note in the CL that we should consider implementing the more deterministic algorithm later. CC @randall77

[ianlancetaylor]

Specifically, the algorithm proposed in #31636 is "find the lexically earliest package [as sorted by import path] that is not initialized yet, but has had all its dependencies initialized, Initialize that package, and repeat."

[griesemer]

@the-zucc The go command passes files to the compiler in lexical file name order.

[apparentlymart]

My sense of this so far has been that init is there as a measure of pragmatism because packages sometimes need to do some work to initialize their own internal state, for rare-but-unavoidable situations where a variable must be initialized by behavior that would not be permitted inline in the declaration.

A significant (but acceptable) consequence of that concession is that because init itself is arbitrary code it can call into other packages that may themselves require initialization, and so it is possible for one package's init to observe the side-effects of another if the ordering is not carefully controlled.

(I will concede that there are some packages in stdlib that seem to be intentionally designed around the existence of initialization side-effects, such as how database/sql deals with "drivers". I personally find that a rather regrettable design choice but it's not worth debating that both because it cannot change now, and because the problem that init functions can observe each other's side-effects remains even if nobody uses this register-on-import pattern.)

It seems to me that the existence of init requires there to be a well-defined inter-package init order in order to avoid dependency hell in larger programs that indirectly depend on packages that are dependent on one another and use init. Thankfully these problems don't seem to arise a lot in practice but when they do they are pretty confusing to debug and unclear how to resolve.

However, I'm very skeptical about the import order of dependencies of a particular package being significant. The author of a package fully controls what it directly depends on, and so can avoid using combinations of packages that don't make sense together, can directly import the packages whose side-effects they depend on, and avoid directly importing packages that contain known bugs (init-order-related or otherwise).

---

I share the same reaction to others in this discussion that having an init function have side-effects visible outside of the process seems like a design error in that package, and that the Go specification need not accommodate or encourage that design error.

However, if we put aside that particular part of the problem then we have an example where the init of "github.com/the-zucc/packagea" depends on the side-effects in the init of "github.com/the-zucc/packageb". If we assume that this interaction is unavoidable for the sake of this discussion, I would expect the one ending in packagea to itself import the one ending in packageb, to represent that it depends on that side-effect.

It feels like a bug to me that packagea depends on an import-time side-effect of packageb without declaring that dependency. With that package dependency declared, I believe the toolchain will already produce the correct ordering overall.

I feel skeptical that the specification needs to guarantee anything stronger than that.

[ianlancetaylor]

@apparentlymart I am definitely sympathetic to that point of view. Still, for a counter-argument, see #31636 (comment) .

[the-zucc]

@the-zucc The go command passes files to the compiler in lexical file name order.

@griesemer Excuse my ignorance, and thank you for the clarification!

@apparentlymart I agree that relying on the import order for side effects being propagated is likely a very bad design choice if we're talking specifically about the example I mentioned earlier. However, wouldn't bringing consistency to import initialization order lessen that fact? Wouldn't it be "okay" (not saying it would be excellent, far from that) to be able to rely on a behavior that we can predict and that is consistent?

Also, the whole idea of relying on import order for side effect propagation, again, starts with the mention in the spec:

To import a package solely for its side-effects (initialization), use the blank identifier as explicit package name:

import _ "lib/math"

This is inconsistent with the very valid principle that everyone has outlined here, that one shouldn't rely on a specific package initialization order for side effect propagation.

In my point of view, it comes back to the same options that were outlined back in 2019 in #31636 (comment), but with a twist: in addition to any decision regarding the options previously outlined, why wouldn't importing for side effects just be discouraged in the spec ? Or does this fall out of the scope of such a specification?

I'm thinking, something can be done that is similar to what is done with dot imports, where they are allowed but discouraged, and it clearly outlines it in the standard linting/syntax highlighting tools for Go in VSCode (staticcheck). This could be the case for imports for side effects. Just throwing that out there.

[rittneje]

This is inconsistent with the vely valid principle that everyone has outlined here, that one shouldn't rely on a specific package initialization order for side effect propagation.

This is inaccurate. You should not rely on a specific initialization order when there is no explicit dependency between the two packages. That is:

1. If package A imports package B, then it is perfectly reasonable for A to rely on B getting fully initialized first.
2. If package A does not import package B, then you should not assume anything about the relative order in which they are initialized. This is not just because the Go specification does not currently prescribe the order, but also for humans it is far easier to reason about what happens when it is only a function of what packages A and B themselves say. If it is contingent upon what some other arbitrary package C says (as is the case in your original proposal based on the order in the import block), then you have to know about the entire application in order to reason about the behavior.

The suggestion to have the language spec mandate that packages be initialized in lexicographic order when there is a "tie" is useful to eliminate currently undefined behavior that could cause grief as the toolchain evolves. However, it would still be ill-advised to rely on this behavior. For example, suppose I have two packages - "mypackage/one" and "mypackage/two". In this hypothetical future, assuming neither imports the other, "mypackage/one" will be initialized first, and "mypackage/two" could implicitly rely on this fact. But then suppose some developer makes changes that cause "mypackage/one" to import "mypackage/two". Now the import order will silently be reversed, causing the application to break in some way. And to the developer making the change, the reason for the failure may not be immediately obvious. If however the original dependency had been made explicit by having "mypackage/two" import "mypackage/one", then the aforementioned change would have resulted in a clear compile-time failure due to the import loop.

[the-zucc]

This is inaccurate. You should not rely on a specific initialization order when there is no explicit dependency between the two packages.

@rittneje If there is an explicit dependency between the packages, the init order between them is predictable, and so your reasoning is not valid for the concerns I am expressing here.

[rittneje]

Order doesn't change if I rename packages within a single module.

Note that in the general case, "renaming a package" and "deleting one package and adding another" are not distinguishable. Thus I don't think this goal is fully achievable without having a way for the developer to define the initialization order outside the package as in the original proposal, which I don't think Go should encourage or support.

For me, both (3) and (4) in your list are essential in order to meet the general expectation that "if I don't change the source code at all, the program still works the same way".

With regards to (1), I feel it is more that the import order is not a good source to base the initialization order on, since (a) it makes it too easy for developers to try to rely on and control it, and (b) it leads to confusion when two different packages (or even two different files in the same package) arrange their imports in contradictory orders.

[rsc]

Right now we do a recursive traversal of the package import graph, with the imports of a given package considered in source code order.

The suggestion here is to completely determine the order, by doing a topological sort where ties are broken by alphabetical order of import path. That makes gofmt's sorting of imports a no-op, which it has never been.

A point was made that renaming a package will change when it gets initialized (in the tiebreaks) but renaming usually results in re-sorting with gofmt too, so it was already changing.

This new order is a valid current order and is completely deterministic and fairly easy to explain. That seems like a win.

[rsc]

This seems like a good idea but maybe we want to do a prototype implementation to convince ourselves there are no hidden gotchas. Does anyone want to try that?

[apparentlymart]

The essence of my earlier reactions was that I worry about a guaranteed initialization order encouraging increased reliance on the (IMHO regrettable) "registration by importing" pattern and possibly other order-sensitive designs like the one that motivated this proposal. I don't relish having more opportunities for unexplained behavior changes as a result of seemingly-unrelated changes in my own packages, particularly if those changes were made quietly and automatically by a tool like goimports.

However, on further reflection I can see that existing code is already effectively relying on the current implementation detail and so it would be undesirable to make the behavior any less predictable than it already is; that would risk breaking systems that are currently working.

Therefore there must be some specific order in each implementation, and if that is true then it's better for all implementations to agree rather than for codebases to behave subtly differently when compiled with a different implementation. Being prescriptive in the specification while making sure the prescription is backward compatible with the main implementation seems like the best way to get there.

Given that, I retract my earlier objections. The specification is a description of how a Go implementation is expected to behave, and is not a "best practices" document describing what is and is not recommended in shared libraries.

I don't have a strong opinion about what exactly the specified ordering is. The proposals above for the details seem fine to me.

[the-zucc]

This seems like a good idea but maybe we want to do a prototype implementation to convince ourselves there are no hidden gotchas. Does anyone want to try that?

@rsc I will try to come up with an implementation, however I can't guarantee a short timeline for it at the moment, simply because of the busy schedule of simultaneous full-time work and college studies.

Being prescriptive in the specification while making sure the prescription is backward compatible with the main implementation seems like the best way to get there.

@apparentlymart very well said !

Last open question about this (hopefully) ; did anyone notice any change in behavior going from an application (with a main function) to just a unit tests file ? Am I wrong in the analysis I outlined in my previous comment here ?

[randall77]

I have a prototype CL for this at https://go-review.googlesource.com/c/go/+/462035

[aclements]

My one minor concern with this is how it interacts with the "plugin" package (or any sort of dynamic loading). Right now, the init order between the main application and any additional packages initialized by loading a plugin is consistent with the underconstrained init order in the spec. I'm not sure any total init order can be consistent with plugin loading (unless the rule accounts for plugins).

Again, this is a minor concern, and the init order of plugins is already inherently different with respect to non-init code, so perhaps we simply don't care.

[rsc]

Are there any objections to this change?

[mdempsky]

Are there any objections to this change?

Making init order more predictable seems desirable, just to reduce unknowns when debugging / minimizing issues. And breaking ties by package path seems as reasonable to me as any other options. (Caveat: The Go spec today only talks about "import paths," not "package paths.")

My only suggestion is since this is backwards compatible with the current specified semantics and there doesn't seem to be any compelling use case for end users to start depending on the new semantics, we could always change the toolchain(s) first and then update the spec (if at all) to match once we're confident the new behavior doesn't introduce any problems of its own.

[rsc]

The important difference between import path and package path arises when using pre-module vendoring as described at https://go.dev/s/go15vendor. It is a good question whether we want to try to explain all that in the spec just to specify the ordering. Since it only comes up with pre-module vendoring, we could just ignore that in the spec and keep saying import path, and understand that people using pre-module vendoring are getting a technical spec violation, but they are essentially opting into an older version of Go anyway.

So I think we could probably edit the spec, say import path, and not worry about the fact that the import path in source code differs from "actual" import path in these rare cases. (The vendoring in the standard library is not rare but it's also not something that affects anyone but us.)

Editing the spec has the benefit of providing something users can rely on, so probably we should if we can.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/462035 mentions this issue: cmd/link: establish dependable package initialization order

[rsc]

Rolled back in https://go.dev/cl/474976 due to problems in Google's internal test suite and perhaps also the aix builder.

[gopherbot]

Change https://go.dev/cl/478916 mentions this issue: cmd/link: establish dependable package initialization order

[randall77]

Retry in CL 478916.

Google internal ran into two issues with this change. I will explain one; the other is very similar.

The problem is a undefined ordering between flags.String (or other variants that make a flag) and flags.Set.

Package a does

package a
import "flag"
var myFlag = flags.String("myFlag", "defaultValue", "description")

Package b does

package b
import "flag"
func init() {
    flags.Set("myFlag", "otherValue")
}

Typically package b is a test package and is modifying the behavior of a for testing purposes. For example, pointing a to a test database instead of the production database.

Critically for the issue at hand, b does not import a directly or indirectly. This is unusual, as normally when b is using the functionality of a it imports a, at least indirectly. But there are cases where even an indirect dependence is missing, like if a registers itself with http.ServeMux.Handle and b then sends a http request to that ServeMux.

Without even an indirect dependence, the initialization order is unspecified (before this CL, that is). If the initialization of a happens first, everything is ok. If the initialization of b happens first, then Set returns an error because the flag isn't defined yet. And then the calling code ignores the returned error, and is blissfully unaware that its Set call didn't do what it expected.

The fundamental problem here is that we're trying to establish dependence by name (a string) and not by an actual language construct reference. It would be better if the only way to set a flag was to get the address of it somehow (which would require exporting it, at least in test mode). In any case, that ship has sailed with how flags.Set is currently defined.

We're working on fixing some of these implicit ordering dependence things inside Google so we can retry the CL. The question here is, how much do we expect external code to run into this? Anything we can do to mitigate it?

One option to consider is having flags.MustSet that panics instead of returning an error.

[ianlancetaylor]

A possibility: if flag.Set is called on a non-existent flag, we record that flag and return an error. If we later see a definition of that flag, we panic. I think this might be OK on the theory that no reasonable program would expect to ignore an error from flag.Set and then later define that flag. That seems like a clear program error.

[gopherbot]

Change https://go.dev/cl/480215 mentions this issue: flag: panic if a flag is defined after being set