Skip to content

spec: define initialization order more precisely #57411

Closed

Description

@the-zucc

Context

Package Initialization

When importing a package, it is initialized according to the sequence described in the Go specification. If that package has dependencies, it makes sense that these are initialized first, and that is what is described in the spec. In many instances, this is very useful and helps to write very succinct code, eliminating much boilerplate code.

Importing for Side Effects

In the Go specification, there is mention that it is possible to import packages for their side-effects:

An import declaration declares a dependency relation between the importing and imported package. It is illegal for a package to import itself, directly or indirectly, or to directly import a package without referring to any of its exported identifiers. To import a package solely for its side-effects (initialization), use the blank identifier as explicit package name:

import _ "lib/math"

Small Issue

Currently, the import order is predictable for packages with import relations. This is very useful and works flawlessly. However, as I've come to realize it, within a series of import statements in a single file, a package that is imported (for its side effects) early in the list of imports has no guarantee of early initialization. Consequently, the initializations are not ran in order, which makes results unpredictable at times.

Example Use Case - Mocking An Execution Environment Before Running Unit Tests

The package names here are not necessarily representative of what has caused this issue, they are just to explain the use case; let's say I have a package, packagea, that initializes a connection to a server, either in its init() or via variable initializers.

package somepackage_test

import _ "github.com/the-zucc/packageb" // this init() function sets an environment variable containing the server URL
import "github.com/the-zucc/packagea" // this package's init() function reads the environment variable

// Just imagine some unit tests here... :)

In this case, I've observed 2 things:

  • There's no guarantee that packageb will be imported and initialized before packagea -- in my case, the package imported the earliest was initialized at the end, effectively not propagating its side effects early enough in the program execution for them to apply to other package initializations, as the environment variable was not set which caused packagea to default to the "production code" server URL, and not the testing URL;
  • There isn't anything in the documentation or in the official Go specification referring to package initialization order within a series of import statements, or within a grouped import.

Proposed Solution

Enable some way of specifying that a package should be imported before any other package for its side effects. As an example, initialization behavior for grouped imports could remain the same, but for separate import statements, statement order could define the initialization order.

import _ "github.com/the-zucc/packageb" // this could be initialized first

import ( // and then the initialization behaviour could remain the same for this grouped import
    "github.com/the-zucc/packagec"
    "github.com/the-zucc/packagea"
)

EDIT: as mentioned in the comments, the proposed solution wouldn't work, unfortunately; however it still gives an idea of what a candidate implementation could look like.

Activity

  1. added this to the Proposal milestone on Dec 20, 2022
  2. changed the title [-]Proposal: Make Package Initialization Order Within a Series of Import Statements Predictable and Controllable[/-] [+]proposal: make package initialization order within a series of import statements predictable and controllable[/+] on Dec 20, 2022
  3. moved this to Incoming in Proposalson Dec 20, 2022
  4. ianlancetaylor commented on Dec 20, 2022

    @ianlancetaylor
    Contributor
  5. seankhliao commented on Dec 20, 2022

    @seankhliao
    Member

    Given the widespread use of goimports and similar tools, the proposed solution won't work as it would be rewritten into a single block with sorted ordering.

    More generally, what if a package does:

    import _ "github.com/the-zucc/packageb" 
    
    import (
        "github.com/the-zucc/packagec"
        "github.com/the-zucc/packagea"
    )
    

    while another does

    import _ "github.com/the-zucc/packagec"
    
    import (
        "github.com/the-zucc/packageb"
        "github.com/the-zucc/packagea"
    )
    

    or even different files in the same package?

  6. seankhliao commented on Dec 20, 2022

    @seankhliao
    Member

    see also discussions in the other direction, pointing to perhaps the more common need to protect against people writing init functions

    removing init #43731
    control over defering init #48174
    import without running init #38450

  7. the-zucc commented on Dec 20, 2022

    @the-zucc
    Author

    Given the widespread use of goimports and similar tools, the proposed solution won't work as it would be rewritten into a single block with sorted ordering.

    Depending on the implementation of this, goimports could need an update anyway.

    More generally, what if a package does:

    import _ "github.com/the-zucc/packageb" 
    
    import (
        "github.com/the-zucc/packagec"
        "github.com/the-zucc/packagea"
    )
    

    while another does

    import _ "github.com/the-zucc/packagec"
    
    import (
        "github.com/the-zucc/packageb"
        "github.com/the-zucc/packagea"
    )
    

    or even different files in the same package?

    That is a great question... couldn't this raise a syntax error ? Like a "pick a lane bud, you're drunk !" 😄

  8. seankhliao commented on Dec 20, 2022

    @seankhliao
    Member

    packaged does the first one, packagee does the second
    you import both packaged and packagee, but not any of packagea, packageb, packagec directly

    are you just never allowed to import these two packages together?

  9. the-zucc commented on Dec 20, 2022

    @the-zucc
    Author

    packaged does the first one, packagee does the second you import both packaged and packagee, but not any of packagea, packageb, packagec directly

    are you just never allowed to import these two packages together?

    Without having thought about it too much (therefore don't take my word for it), I think I would simply raise a syntax error on both import lines (for packaged and packagee...) There is hardly any reason why one would want to have 2 different import orders, and if there was a reason, that reason doesn't exist as of today, because at the moment, package initialization order isn't guaranteed (and so we can eliminate that problem by raising a syntax error.)

  10. the-zucc commented on Dec 20, 2022

    @the-zucc
    Author

    There is hardly any reason why one would want to have 2 different import orders, and if there was a reason, that reason doesn't exist as of today, because at the moment, package initialization order isn't guaranteed (and so we can eliminate that problem by raising a syntax error.)

    Actually, I see a potential problem with applications with huge dependency trees with open source packages. One would want to take this into consideration if something is to be done...

  11. rittneje commented on Dec 20, 2022

    @rittneje
    Contributor

    You should not connect to a server from init() as that violates the concept of writing testable code, as you have encountered.

    Instead, packagea should have a proper constructor that gets invoked. Each unit test is then responsible for calling it. This establishes proper isolation between each test case, which is a very useful property.

    Meanwhile, unit tests for other packages that depend on packagea should likely be using some interface instead of direct function calls, which allows for mocking.

  12. griesemer commented on Dec 20, 2022

    @griesemer
    Contributor

    See also #31636. You can force the init order by ordering the imports accordingly and into "blocks" so that they don't get re-ordered (example). Note that this is not prescribed by the language, it's an implementation behavior.

    That said, it is much more robust to be explicit and call initialization functions explicitly if they cause side effects.

  13. the-zucc commented on Dec 20, 2022

    @the-zucc
    Author

    You should not connect to a server from init() as that violates the concept of writing testable code, as you have encountered.

    I guess you are right, there are probably other ways to get succint code without connecting to a server in the init() function. However the fundamental question regarding side effects is still of interest, for other use cases.

  14. 66 remaining items

  15. randall77 commented on Nov 29, 2023

    @randall77
    Contributor

    If you want packageB to be initialized before packageA, then do, in packageA

    import _ "issue/packageB"
    
  16. hunshcn commented on Nov 29, 2023

    @hunshcn

    If you want packageB to be initialized before packageA, then do, in packageA

    import _ "issue/packageB"
    

    This is the simplest case. In a normal scenario, packageA/B may be a third-party package.

  17. randall77 commented on Nov 29, 2023

    @randall77
    Contributor

    Can you be specific about the two third-party packages you're having trouble with?

  18. hunshcn commented on Nov 29, 2023

    @hunshcn

    Can you be specific about the two third-party packages you're having trouble with?

    I don't think the specific package is critical. In go1.20, adjusting the import order in main can affect initialization order. go1.20 cannot

    But for your question, I will also make an example to describe the scene. Please give me some time.

  19. hunshcn commented on Nov 29, 2023

    @hunshcn

    @randall77 Thank You for Your Time

    in my project (a k8s controller)

    there are two workqueue metric registration, "sigs.k8s.io/controller-runtime/pkg/metrics" and "k8s.io/component-base/metrics/prometheus/workqueue".

    The simultaneous import of the two packages is not explicitly declared by me, I just want to use "sigs.k8s.io/controller-runtime/pkg/metrics". "k8s.io/component-base/metrics/prometheus/workqueue" from the deep dependency

    They both registry workqueue metric, but workqueue provider can only registry once. So the import order becomes very important.

    I have no idea to deal with this problem on go 1.21 at present. In go 1.20, I only need to import "sigs.k8s.io/controller-runtime/pkg/metrics" at the top of package main to solve it.

  20. randall77 commented on Nov 29, 2023

    @randall77
    Contributor

    I'm a bit confused by your answer. You quote two packages that you want to initialize in a specific order, but I don't understand why you need them initialized in a specific order.

    My best guess: are they both calling into k8s.io/component-base/metrics/legacyregistry.MustRegister, and the ordering of those calls matters?

  21. hunshcn commented on Nov 29, 2023

    @hunshcn

    @randall77 You are very wise. Just an hour ago,I located the citation exception through dependency analysis. Two years ago, our team members accidentally imported k8s.io/apiserver/pkg/server, and after a lengthy chain import, "k8s.io/component-base/metrics/promises/work queue" was mistakenly imported. This erroneous introduction was not discovered until this change. (Although this problem was discovered before, it was not deeply located, and adjusting the import order was enough to solve the problem.)

    However, this is also an example, and defining initialization order may be a potential requirement.

  22. randall77 commented on Nov 29, 2023

    @randall77
    Contributor

    I think I still need more details to understand.
    What, exactly, is the failure you are seeing? What command did you run, what did you expect as a result, and what did you see instead?

    @randall77 You are very wise.

    Does that mean I'm right about legacyregistery.MustRegister? If that's the case, I still don't understand the problem. What about that registration is order-dependent?

    Just an hour ago,I located the citation exception through dependency analysis.

    I don't understand what this means.

    Two years ago, our team members accidentally imported k8s.io/apiserver/pkg/server, and after a lengthy chain import, "k8s.io/component-base/metrics/promises/work queue" was mistakenly imported. This erroneous introduction was not discovered until this change.

    By "this change", you mean the one you're currently working on? So there was a latent bug for 2 years that you're now tripping up on? Shouldn't that latent bug be fixed instead of papering back over it by reordering initialization?

  23. hunshcn commented on Nov 29, 2023

    @hunshcn

    By "this change", you mean the one you're currently working on? So there was a latent bug for 2 years that you're now tripping up on? Shouldn't that latent bug be fixed instead of papering back over it by reordering initialization?

    No, after discovering what I said, I fixed it.

    I mean, similar things may happen in other scenes. Of course, I provided a bad example. I'm sorry.

  24. randall77 commented on Nov 29, 2023

    @randall77
    Contributor

    Ok, thanks for the discussion.
    I think we're in agreement that there could be situations where initialization order matters and people would want to control it. But barring an actual example where this happens, it is hard to justify changing the current semantics at this point. Particularly, I would want to see a situation where there isn't some other means of achieving the desired result.

  25. removed this from Proposalson Apr 18, 2024
  26. locked and limited conversation to collaborators on Nov 28, 2024
  27. added 2 commits that reference this issue on Sep 29, 2026
    ce2a609
    2b92c39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions