Repository navigation
Description
Context
Package Initialization
When importing a package, it is initialized according to the sequence described in the Go specification. If that package has dependencies, it makes sense that these are initialized first, and that is what is described in the spec. In many instances, this is very useful and helps to write very succinct code, eliminating much boilerplate code.
Importing for Side Effects
In the Go specification, there is mention that it is possible to import packages for their side-effects:
An import declaration declares a dependency relation between the importing and imported package. It is illegal for a package to import itself, directly or indirectly, or to directly import a package without referring to any of its exported identifiers. To import a package solely for its side-effects (initialization), use the blank identifier as explicit package name:
import _ "lib/math"
Small Issue
Currently, the import order is predictable for packages with import relations. This is very useful and works flawlessly. However, as I've come to realize it, within a series of import statements in a single file, a package that is imported (for its side effects) early in the list of imports has no guarantee of early initialization. Consequently, the initializations are not ran in order, which makes results unpredictable at times.
Example Use Case - Mocking An Execution Environment Before Running Unit Tests
The package names here are not necessarily representative of what has caused this issue, they are just to explain the use case; let's say I have a package, packagea, that initializes a connection to a server, either in its init() or via variable initializers.
package somepackage_test
import _ "github.com/the-zucc/packageb" // this init() function sets an environment variable containing the server URL
import "github.com/the-zucc/packagea" // this package's init() function reads the environment variable
// Just imagine some unit tests here... :)In this case, I've observed 2 things:
- There's no guarantee that
packagebwill be imported and initialized beforepackagea-- in my case, the package imported the earliest was initialized at the end, effectively not propagating its side effects early enough in the program execution for them to apply to other package initializations, as the environment variable was not set which causedpackageato default to the "production code" server URL, and not the testing URL; - There isn't anything in the documentation or in the official Go specification referring to package initialization order within a series of import statements, or within a grouped import.
Proposed Solution
Enable some way of specifying that a package should be imported before any other package for its side effects. As an example, initialization behavior for grouped imports could remain the same, but for separate import statements, statement order could define the initialization order.
import _ "github.com/the-zucc/packageb" // this could be initialized first
import ( // and then the initialization behaviour could remain the same for this grouped import
"github.com/the-zucc/packagec"
"github.com/the-zucc/packagea"
)EDIT: as mentioned in the comments, the proposed solution wouldn't work, unfortunately; however it still gives an idea of what a candidate implementation could look like.
Activity
[-]Proposal: Make Package Initialization Order Within a Series of Import Statements Predictable and Controllable[/-][+]proposal: make package initialization order within a series of import statements predictable and controllable[/+]ianlancetaylor commentedon Dec 20, 2022
CC @griesemer
seankhliao commentedon Dec 20, 2022
Given the widespread use of
goimportsand similar tools, the proposed solution won't work as it would be rewritten into a single block with sorted ordering.More generally, what if a package does:
while another does
or even different files in the same package?
seankhliao commentedon Dec 20, 2022
see also discussions in the other direction, pointing to perhaps the more common need to protect against people writing init functions
removing init #43731
control over defering init #48174
import without running init #38450
the-zucc commentedon Dec 20, 2022
Depending on the implementation of this,
goimportscould need an update anyway.That is a great question... couldn't this raise a syntax error ? Like a "pick a lane bud, you're drunk !" 😄
seankhliao commentedon Dec 20, 2022
packaged does the first one, packagee does the second
you import both packaged and packagee, but not any of packagea, packageb, packagec directly
are you just never allowed to import these two packages together?
the-zucc commentedon Dec 20, 2022
Without having thought about it too much (therefore don't take my word for it), I think I would simply raise a syntax error on both import lines (for packaged and packagee...) There is hardly any reason why one would want to have 2 different import orders, and if there was a reason, that reason doesn't exist as of today, because at the moment, package initialization order isn't guaranteed (and so we can eliminate that problem by raising a syntax error.)
the-zucc commentedon Dec 20, 2022
Actually, I see a potential problem with applications with huge dependency trees with open source packages. One would want to take this into consideration if something is to be done...
rittneje commentedon Dec 20, 2022
You should not connect to a server from
init()as that violates the concept of writing testable code, as you have encountered.Instead, packagea should have a proper constructor that gets invoked. Each unit test is then responsible for calling it. This establishes proper isolation between each test case, which is a very useful property.
Meanwhile, unit tests for other packages that depend on packagea should likely be using some interface instead of direct function calls, which allows for mocking.
griesemer commentedon Dec 20, 2022
See also #31636. You can force the init order by ordering the imports accordingly and into "blocks" so that they don't get re-ordered (example). Note that this is not prescribed by the language, it's an implementation behavior.
That said, it is much more robust to be explicit and call initialization functions explicitly if they cause side effects.
the-zucc commentedon Dec 20, 2022
I guess you are right, there are probably other ways to get succint code without connecting to a server in the init() function. However the fundamental question regarding side effects is still of interest, for other use cases.
66 remaining items
randall77 commentedon Nov 29, 2023
If you want packageB to be initialized before packageA, then do, in packageA
hunshcn commentedon Nov 29, 2023
This is the simplest case. In a normal scenario, packageA/B may be a third-party package.
randall77 commentedon Nov 29, 2023
Can you be specific about the two third-party packages you're having trouble with?
hunshcn commentedon Nov 29, 2023
I don't think the specific package is critical. In go1.20, adjusting the import order in main can affect initialization order. go1.20 cannot
But for your question, I will also make an example to describe the scene. Please give me some time.
hunshcn commentedon Nov 29, 2023
@randall77 Thank You for Your Time
in my project (a k8s controller)
there are two workqueue metric registration, "sigs.k8s.io/controller-runtime/pkg/metrics" and "k8s.io/component-base/metrics/prometheus/workqueue".
The simultaneous import of the two packages is not explicitly declared by me, I just want to use "sigs.k8s.io/controller-runtime/pkg/metrics". "k8s.io/component-base/metrics/prometheus/workqueue" from the deep dependency
They both registry workqueue metric, but workqueue provider can only registry once. So the import order becomes very important.
I have no idea to deal with this problem on go 1.21 at present. In go 1.20, I only need to import "sigs.k8s.io/controller-runtime/pkg/metrics" at the top of package main to solve it.
randall77 commentedon Nov 29, 2023
I'm a bit confused by your answer. You quote two packages that you want to initialize in a specific order, but I don't understand why you need them initialized in a specific order.
My best guess: are they both calling into
k8s.io/component-base/metrics/legacyregistry.MustRegister, and the ordering of those calls matters?hunshcn commentedon Nov 29, 2023
@randall77 You are very wise. Just an hour ago,I located the citation exception through dependency analysis. Two years ago, our team members accidentally imported
k8s.io/apiserver/pkg/server, and after a lengthy chain import, "k8s.io/component-base/metrics/promises/work queue" was mistakenly imported. This erroneous introduction was not discovered until this change. (Although this problem was discovered before, it was not deeply located, and adjusting the import order was enough to solve the problem.)However, this is also an example, and defining initialization order may be a potential requirement.
randall77 commentedon Nov 29, 2023
I think I still need more details to understand.
What, exactly, is the failure you are seeing? What command did you run, what did you expect as a result, and what did you see instead?
Does that mean I'm right about
legacyregistery.MustRegister? If that's the case, I still don't understand the problem. What about that registration is order-dependent?I don't understand what this means.
By "this change", you mean the one you're currently working on? So there was a latent bug for 2 years that you're now tripping up on? Shouldn't that latent bug be fixed instead of papering back over it by reordering initialization?
hunshcn commentedon Nov 29, 2023
No, after discovering what I said, I fixed it.
I mean, similar things may happen in other scenes. Of course, I provided a bad example. I'm sorry.
randall77 commentedon Nov 29, 2023
Ok, thanks for the discussion.
I think we're in agreement that there could be situations where initialization order matters and people would want to control it. But barring an actual example where this happens, it is hard to justify changing the current semantics at this point. Particularly, I would want to see a situation where there isn't some other means of achieving the desired result.
cmd/link: establish dependable package initialization order
cmd/link: establish dependable package initialization order