Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NET-6741] make: Add target for updating dependencies across all modules #19785

Merged
merged 1 commit into from
Feb 21, 2024
Merged

[NET-6741] make: Add target for updating dependencies across all modules #19785

merged 1 commit into from
Feb 21, 2024

Conversation

zalimeni
Copy link
Member

@zalimeni zalimeni commented Nov 30, 2023 •
edited
Loading

To enable more consistent and error-proof dependency management, add a Make target that will set a dependency version across all submodules that require it.

Also runs go mod tidy. This first ensures the dependency addition is reverted if the module in question does not require it; it also ensures that any additional cleanup needed in go.mod/go.sum is applied.

Note on approach vs. go.work

At one point, it seemed we could avoid executing go get per each go.mod by instead moving to a versioned go.work and using workspace-based dependency management commands. Though there's still reasons to move to go.work beyond this change, it probably won't work for bumping dependencies conveniently: go work sync consolidates versions using MVS, so in practice, it seems we still need to use a visit-all approach to be able to blindly bump a given dependency only where it’s used. go work sync simply enforces consistency across all go.mod files once any go.mod is updated (docs):

The syncing is done by sequentially upgrading each of the dependency modules specified in a workspace module to the version in the build list if the dependency module's version is not already the same as the build list's version. Note that Minimal Version Selection guarantees that the build list's version of each module is always the same or higher than that in each workspace module.

Description

Simplify dependency management, particularly for addressing CVEs where a consistent minimum version is required.

Example run:

❯ make go-mod-get DEP_VERSION=github.com/hashicorp/raft@v1.6.0
--> Running go get github.com/hashicorp/raft@v1.6.0 (.)
go: upgraded github.com/hashicorp/go-msgpack/v2 v2.0.0 => v2.1.1
go: upgraded github.com/hashicorp/raft v1.5.0 => v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/sync v0.3.0 => v0.4.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (.)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./api)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./api)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./envoyextensions)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./envoyextensions)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./internal/tools/proto-gen-rpc-glue)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
go: added golang.org/x/sys v0.13.0
--> Running go mod tidy (./internal/tools/proto-gen-rpc-glue)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./internal/tools/protoc-gen-consul-rate-limit)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
--> Running go mod tidy (./internal/tools/protoc-gen-consul-rate-limit)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./proto-public)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./proto-public)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./sdk)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./sdk)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test-integ)
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (./test-integ)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/connect/envoy/test-sds-server)
go: upgraded github.com/hashicorp/raft v1.4.0 => v1.6.0
--> Running go mod tidy (./test/integration/connect/envoy/test-sds-server)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/consul-container)
go: upgraded github.com/hashicorp/raft v1.5.0 => v1.6.0
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/sync v0.3.0 => v0.4.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (./test/integration/consul-container)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: added golang.org/x/sys v0.13.0
--> Running go mod tidy (./test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./testing/deployer)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./testing/deployer)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./troubleshoot)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./troubleshoot)

❯ git status
On branch zalimeni/add-make-target-dependency-update-modules
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   api/go.mod
	modified:   api/go.sum
	modified:   envoyextensions/go.mod
	modified:   envoyextensions/go.sum
	modified:   go.mod
	modified:   go.sum
	modified:   internal/tools/proto-gen-rpc-glue/go.mod
	modified:   internal/tools/proto-gen-rpc-glue/go.sum
	modified:   proto-public/go.mod
	modified:   proto-public/go.sum
	modified:   sdk/go.mod
	modified:   sdk/go.sum
	modified:   test-integ/go.mod
	modified:   test-integ/go.sum
	modified:   test/integration/connect/envoy/test-sds-server/go.mod
	modified:   test/integration/connect/envoy/test-sds-server/go.sum
	modified:   test/integration/consul-container/go.mod
	modified:   test/integration/consul-container/go.sum
	modified:   test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files/go.mod
	modified:   test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files/go.sum
	modified:   testing/deployer/go.mod
	modified:   testing/deployer/go.sum
	modified:   troubleshoot/go.mod
	modified:   troubleshoot/go.sum

PR Checklist

  • updated test coverage
  • external facing docs updated
  • appropriate backport labels added
  • not a security concern

@zalimeni zalimeni added pr/no-changelog PR does not need a corresponding .changelog entry backport/1.15 This release series is no longer active on CE. Use backport/ent/1.15. backport/1.16 This release series is no longer active on CE. Use backport/ent/1.16. backport/1.17 This release series is no longer active on CE. Use backport/ent/1.17. labels Nov 30, 2023
@zalimeni
Copy link
Member Author

After talking w/ Matt K., we think go.work would be a nice solution w/ added benefits for replace directive removal, so going to give that a shot intead.

@zalimeni zalimeni closed this Nov 30, 2023
@zalimeni
Copy link
Member Author

Update: go work sync consolidates versions using MVS, so in practice, it seems we still need to use a visit-all approach to be able to blindly bump a given dependency only where it’s used. go work sync simply enforces consistency across all go.mod files once any go.mod is updated (docs):

The syncing is done by sequentially upgrading each of the dependency modules specified in a workspace module to the version in the build list if the dependency module's version is not already the same as the build list's version. Note that Minimal Version Selection guarantees that the build list's version of each module is always the same or higher than that in each workspace module.

@zalimeni zalimeni reopened this Feb 21, 2024
@zalimeni
make: Add target for updating dependencies across all modules
6f0e24c 
To enable more consistent and error-proof dependency management, add a
Make target that will set a dependency version across all submodules
that require it.

Also runs `go mod tidy`. This first ensures the dependency addition is
reverted if the module in question does not require it; it also ensures
that any additional cleanup needed in `go.mod`/`go.sum` is applied.
@zalimeni zalimeni force-pushed the zalimeni/net-6741-add-make-target-dependency-update-modules branch from c0704a6 to 6f0e24c Compare February 21, 2024 14:27
@zalimeni zalimeni marked this pull request as ready for review February 21, 2024 14:41
@zalimeni zalimeni merged commit 858691b into main Feb 21, 2024
89 of 90 checks passed
@zalimeni zalimeni deleted the zalimeni/net-6741-add-make-target-dependency-update-modules branch February 21, 2024 16:11
This was referenced Feb 21, 2024
Closed
Merged
Merged
Merged
This was referenced Feb 21, 2024
Merged
Merged
This was referenced Feb 21, 2024
Merged
Merged
Merged
Merged
jmurret pushed a commit that referenced this pull request Feb 21, 2024
@zalimeni @jmurret
[NET-6741] make: Add target for updating dependencies across all modu…
6840da9 
…les (#19785)

make: Add target for updating dependencies across all modules

To enable more consistent and error-proof dependency management, add a
Make target that will set a dependency version across all submodules
that require it.

Also runs `go mod tidy`. This first ensures the dependency addition is
reverted if the module in question does not require it; it also ensures
that any additional cleanup needed in `go.mod`/`go.sum` is applied.
@zalimeni zalimeni mentioned this pull request Feb 21, 2024
Merged
This was referenced Feb 21, 2024
Merged
Merged
Merged
Merged
@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

5 similar comments
@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Contributor

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@jmurret jmurret added backport/1.15 This release series is no longer active on CE. Use backport/ent/1.15. and removed backport/1.15 This release series is no longer active on CE. Use backport/ent/1.15. labels Feb 27, 2024
@hc-github-team-consul-core hc-github-team-consul-core mentioned this pull request Feb 27, 2024
Closed
4 tasks
This was referenced Sep 27, 2024
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mkeeler mkeeler mkeeler approved these changes

@johnlanda johnlanda Awaiting requested review from johnlanda

Assignees
No one assigned
Labels
backport/1.15 This release series is no longer active on CE. Use backport/ent/1.15. backport/1.16 This release series is no longer active on CE. Use backport/ent/1.16. backport/1.17 This release series is no longer active on CE. Use backport/ent/1.17. pr/no-changelog PR does not need a corresponding .changelog entry pr/no-metrics-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zalimeni @hc-github-team-consul-core @mkeeler @jmurret