fix: allow projects in separate directories to run in parallel #2131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the release notes for v0.13.0: https://github.com/runatlantis/atlantis/blob/master/CHANGELOG.md#features-4 > Running in parallel is only supported if you're using workspaces to separate your projects. Projects in separate directories can not be run in parallel currently. This commit adds `path` as an argument to `DefaultWorkingDirLocker.TryLock()` and includes `path` in the `workspaceKey` used to check if a project is locked. This should allow projects in separate directories to run in parallel. To my knowledge, there is no functional reason that projects in separate directories cannot run in parallel. All calls to `DefaultWorkingDirLocker.TryLock()` have been updated. A new unit test `TestTryLockWithDifferentPaths` has been added to test the behavior of locking two separate directories with the same workspace name.
KevinSnyderCodes
changed the title
|
Looks good to me, could you add maybe a doc with an example usage? |
jamengual
approved these changes
Mar 21, 2022
jamengual
approved these changes
Mar 21, 2022
|
Is there an ETA for cutting a new release with this in it? |
KevinSnyderCodes
pushed a commit
to KevinSnyderCodes/atlantis
that referenced
this pull request
Apr 4, 2022
`WorkingDirLocker` and `WorkingDir` are closely related -- the former acquires a lock, which ensures that the latter can safely operate on a given file path. In runatlantis#2131 we added `path` to the `WorkingDirLocker` lock key, but neglected to add the same to `WorkingDir`. This commit adds `path` as an argument to certain `WorkingDir` methods, and includes `path` in the directory that we use to clone the repository for a given project. Since `path` can include certain special characters such as `/`, we encode `path` as base32 when using it as part of a file path. This should ensure no special characters are used in the filesystem, and that the value can be decoded if desired (unlike hashes such as md5). Additional changes: - All calls to changed methods have been updated, including unit tests - Mocks have been regenerated for `WorkingDir` and `WorkingDirLocker` - `working_dir_test.go` - Commands that operate on filesystem paths have been updated to include the base32 encoded `path` value - When running `git init`, we include `-b master` as additional arguments, to ensure that `master` is our default branch (expected by the unit tests)
jamengual
pushed a commit
that referenced
this pull request
Apr 8, 2022
* fix: add path to WorkingDir methods `WorkingDirLocker` and `WorkingDir` are closely related -- the former acquires a lock, which ensures that the latter can safely operate on a given file path. In #2131 we added `path` to the `WorkingDirLocker` lock key, but neglected to add the same to `WorkingDir`. This commit adds `path` as an argument to certain `WorkingDir` methods, and includes `path` in the directory that we use to clone the repository for a given project. Since `path` can include certain special characters such as `/`, we encode `path` as base32 when using it as part of a file path. This should ensure no special characters are used in the filesystem, and that the value can be decoded if desired (unlike hashes such as md5). Additional changes: - All calls to changed methods have been updated, including unit tests - Mocks have been regenerated for `WorkingDir` and `WorkingDirLocker` - `working_dir_test.go` - Commands that operate on filesystem paths have been updated to include the base32 encoded `path` value - When running `git init`, we include `-b master` as additional arguments, to ensure that `master` is our default branch (expected by the unit tests) * Try fixing E2E tests * Fix DefaultPendingPlanFinder In addition to iterating over `workspaceDirs`, also iterate over `pathDirs`, and use both `workspace` and `path` to build `repoDir`. * Fix DefaultPendingPlanFinder unit tests * Fix DefaultProjectCommandBuilder unit tests Co-authored-by: Kevin Snyder <kevinsnyder@KevinSnydersMBP.lan> Co-authored-by: Kevin Snyder <kevinsnyder@ip-192-168-1-188.ec2.internal>
|
I think maybe this introduces a bug? All of my PRs now start with error: Then similar during apply: But the other workspaces continue. There is no |
krrrr38
pushed a commit
to krrrr38/atlantis
that referenced
this pull request
Dec 16, 2022
…lantis#2131) * fix: add path to DefaultWorkingDirLocker.TryLock() In the release notes for v0.13.0: https://github.com/runatlantis/atlantis/blob/master/CHANGELOG.md#features-4 > Running in parallel is only supported if you're using workspaces to separate your projects. Projects in separate directories can not be run in parallel currently. This commit adds `path` as an argument to `DefaultWorkingDirLocker.TryLock()` and includes `path` in the `workspaceKey` used to check if a project is locked. This should allow projects in separate directories to run in parallel. To my knowledge, there is no functional reason that projects in separate directories cannot run in parallel. All calls to `DefaultWorkingDirLocker.TryLock()` have been updated. A new unit test `TestTryLockWithDifferentPaths` has been added to test the behavior of locking two separate directories with the same workspace name. * Add documntation for parallel_plan and parallel_apply options Co-authored-by: Kevin Snyder <kevinsnyder@ip-192-168-4-61.ec2.internal> Co-authored-by: Kevin Snyder <kevinsnyder@ip-10-60-10-94.ec2.internal>
krrrr38
pushed a commit
to krrrr38/atlantis
that referenced
this pull request
Dec 16, 2022
* fix: add path to WorkingDir methods `WorkingDirLocker` and `WorkingDir` are closely related -- the former acquires a lock, which ensures that the latter can safely operate on a given file path. In runatlantis#2131 we added `path` to the `WorkingDirLocker` lock key, but neglected to add the same to `WorkingDir`. This commit adds `path` as an argument to certain `WorkingDir` methods, and includes `path` in the directory that we use to clone the repository for a given project. Since `path` can include certain special characters such as `/`, we encode `path` as base32 when using it as part of a file path. This should ensure no special characters are used in the filesystem, and that the value can be decoded if desired (unlike hashes such as md5). Additional changes: - All calls to changed methods have been updated, including unit tests - Mocks have been regenerated for `WorkingDir` and `WorkingDirLocker` - `working_dir_test.go` - Commands that operate on filesystem paths have been updated to include the base32 encoded `path` value - When running `git init`, we include `-b master` as additional arguments, to ensure that `master` is our default branch (expected by the unit tests) * Try fixing E2E tests * Fix DefaultPendingPlanFinder In addition to iterating over `workspaceDirs`, also iterate over `pathDirs`, and use both `workspace` and `path` to build `repoDir`. * Fix DefaultPendingPlanFinder unit tests * Fix DefaultProjectCommandBuilder unit tests Co-authored-by: Kevin Snyder <kevinsnyder@KevinSnydersMBP.lan> Co-authored-by: Kevin Snyder <kevinsnyder@ip-192-168-1-188.ec2.internal>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the release notes for v0.13.0: https://github.com/runatlantis/atlantis/blob/master/CHANGELOG.md#features-4
This PR adds
pathas an argument toDefaultWorkingDirLocker.TryLock()and includespathin theworkspaceKeyused to check if a project is locked. This allows projects in separate directories to run in parallel.To my knowledge, there is no functional reason against or unintended side effects from projects in separate directories running in parallel.
Additional changes:
DefaultWorkingDirLocker.TryLock()WorkingDirLockerinterface and all implementationsTestTryLockWithDifferentPathsthat tests the behavior of locking two separate directories with the same workspace name