feat: Add project depends on functionality #3292

Luay-Sol · 2023-04-04T14:23:33Z

what

The main purpose of this PR is to add Dependencies functionality to Atlantis. The change introduces a new key depends_on to the project config in the atlantis.yaml file.

why

Using the depends_on functionality will allow Atlantis to handle cases requiring one project to be applied prior to the other. In other words, allow promotions between projects/environments (dev -> staging -> production)

tests

I have tested my changes by adding new unit tests, running Atlantis locally, and testing the depends_on functionality.

references

Somehow related to this feature request: Support prerequisite apply of projects for apply_requirements #391, but not exactly. The implementation followed in this PR does not depend on the apply requirement key.

server/core/config/valid/global_cfg.go

jamengual · 2023-04-08T06:09:44Z

server/events/command/project_context.go

@@ -62,6 +69,8 @@ type ProjectContext struct {
 	PullReqStatus models.PullReqStatus
 	// CurrentProjectPlanStatus is the status of the current project prior to this command.
 	ProjectPlanStatus models.ProjectPlanStatus
+


could you add a comment here of what this is doing?

jamengual · 2023-04-08T06:11:53Z

will these work by running atlantis plan and then atlantis apply in all projects and the apply will go though the dependencies in order?

jamengual · 2023-04-08T06:12:25Z

Thanks for the contribution @Luay-Sol

Luay-Sol · 2023-04-14T15:26:24Z

will these work by running atlantis plan and then atlantis apply in all projects and the apply will go though the dependencies in order?

Actually, this feature is more for the case where the user tries to apply to a specific project/workspace. What you have described already exists in Atlantis and is called execution_order_group. So I see this feature will work in conjunction with the execution_order_group. For example, this is how my project block looks like:

projects:
  - name: development
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/development.tfvars"]
    execution_order_group: 1
    workspace: development
    workflow: infra
  - name:staging
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/staging.tfvars"]
    depends_on: ["development"]
    execution_order_group: 2
    workspace: staging
    workflow: infra
  - name: production
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/production.tfvars"]
    depends_on: ["staging"]
    execution_order_group: 3
    workspace: production
    workflow: infra

So with this, if the user just comments atlantis apply the execution_order_group will ensure that the changes will go development->staging->production. Else if the user makes the apply command sequentially atlantis apply -w development, then atlantis apply -w staging, then atlantis apply -w production, the depends_on feature will make sure that production is not applied before staging for example.

jamengual · 2023-04-14T20:18:33Z

Amazing that is what I was expecting and I can see how useful this will be.

GenPage

Great work. Thank you for your contribution. Please address nitpick comments. I envision we can deprecate execution_order_group after this. It's a bit ambiguous. I feel like overloading depends_on is a better place. If you need the projects executed in a particular order, that implies dependency.

I can see needing only "ordering" vs a "hard fail" like dependencies. How difficult would it be to add a key somewhat like:

version: 3
    projects:
    - dir: .
      name: project-2
      depends_on: 
           softFail: false
           projects: [project-1]

Thoughts?

GenPage · 2023-04-14T21:04:50Z

server/core/config/raw/project.go

@@ -31,6 +31,7 @@ type Project struct {
 	PlanRequirements          []string  `yaml:"plan_requirements,omitempty"`
 	ApplyRequirements         []string  `yaml:"apply_requirements,omitempty"`
 	ImportRequirements        []string  `yaml:"import_requirements,omitempty"`
+	Dependencies              []string  `yaml:"depends_on,omitempty"`


nitpick: Can we follow the annotation style and rename the variable to DependsOn

GenPage · 2023-04-14T21:05:18Z

server/core/config/raw/project.go

@@ -72,12 +73,17 @@ func (p Project) Validate() error {
 		return errors.Wrapf(err, "parsing: %s", branch)
 	}

+	Dependencies := func(value interface{}) error {


nitpick: Can we follow the annotation style and rename the variable to DependsOn

GenPage · 2023-04-14T21:05:23Z

server/core/config/raw/project.go

 	return validation.ValidateStruct(&p,
 		validation.Field(&p.Dir, validation.Required, validation.By(hasDotDot)),
 		validation.Field(&p.PlanRequirements, validation.By(validPlanReq)),
 		validation.Field(&p.ApplyRequirements, validation.By(validApplyReq)),
 		validation.Field(&p.ImportRequirements, validation.By(validImportReq)),
 		validation.Field(&p.TerraformVersion, validation.By(VersionValidator)),
+		validation.Field(&p.Dependencies, validation.By(Dependencies)),


nitpick: Can we follow the annotation style and rename the variable to DependsOn

GenPage · 2023-04-14T21:05:33Z

server/core/config/raw/project.go

@@ -121,6 +127,8 @@ func (p Project) ToValid() valid.Project {

 	v.Name = p.Name

+	v.Dependencies = p.Dependencies


nitpick: Can we follow the annotation style and rename the variable to DependsOn

GenPage · 2023-04-14T21:05:53Z

server/core/config/valid/global_cfg.go

@@ -88,6 +85,7 @@ type MergedProjectCfg struct {
 	ImportRequirements        []string
 	Workflow                  Workflow
 	AllowedWorkflows          []string
+	Dependencies              []string


Rename to DependsOn

server/core/config/valid/global_cfg.go

GenPage · 2023-04-14T22:41:33Z

server/core/config/valid/repo_cfg.go

@@ -129,6 +129,7 @@ type Project struct {
 	PlanRequirements          []string
 	ApplyRequirements         []string
 	ImportRequirements        []string
+	Dependencies              []string


Rename to DependsOn

GenPage · 2023-04-14T22:41:58Z

server/events/command/project_context.go

+	// Atlantis uses this information to valid the apply
+	// orders and to warn the user if they're applying a project that
+	// depends on other projects.
+	Dependencies []string


Rename to DependsOn

GenPage · 2023-04-14T22:44:29Z

server/events/command_requirement_handler_test.go

+				},
+			},
+			wantFailure: "Can't apply your project unless you apply its dependencies: [project2]",
+			wantErr:     assert.NoError,


nitpick: Tests are good so far, can we include some more scenarios that include 3 projects to make sure Atlantis doesn't continue past the 2nd project?

Project 1: success
Project 2: fail
Project 3: skipped

@Luay-Sol - re: the bug you were struggling with, is there a test case in here that ensures no regressions against that bug? (in case I change the depends_on code)

oh, so the bug was related to missing

&& project.Status != models.PlannedNoChangesPlanStatus

to break the dependency chain, and I see above you already added a test case for this 🎉

GenPage · 2023-04-14T22:45:47Z

server/events/command_requirement_handler.go

@@ -65,6 +67,20 @@ func (a *DefaultCommandRequirementHandler) ValidateApplyProject(repoDir string,
 	return "", nil
 }

+func (a *DefaultCommandRequirementHandler) ValidateProjectDependencies(ctx command.ProjectContext) (failure string, err error) {


Dependencies makes sense here for the function name. I know I mentioned renaming some of the variables to DependsOn. I'm not going to nit pick on every single one, just use good judgment on which ones make sense to change.

nitrocode · 2023-04-18T13:55:21Z

@GenPage

I envision we can deprecate execution_order_group after this

From @Luay-Sol 's above comment, it seems like we can use both execution order group and depends_on together, no?

Luay-Sol · 2023-04-18T15:59:10Z

Great work. Thank you for your contribution. Please address nitpick comments. I envision we can deprecate execution_order_group after this. It's a bit ambiguous. I feel like overloading depends_on is a better place. If you need the projects executed in a particular order, that implies dependency.

I can see needing only "ordering" vs a "hard fail" like dependencies. How difficult would it be to add a key somewhat like:
version: 3
    projects:
    - dir: .
      name: project-2
      depends_on: 
           softFail: false
           projects: [project-1]
Thoughts?

Thanks for your feedback @GenPage. For your first point about deprecating execution_order_group, I agree and I think in the next iteration we can beef up depends_on and have it handle both cases (explicit project/workspace apply & global apply).

For your second point, It is not complicated to add such a key. However, I would like to better understand the use case. I am trying to figure out when a softfail will be used. If it is just to prevent failure, maybe not define any dependencies to the project.

Thanks.

Luay-Sol · 2023-04-18T16:05:43Z

@GenPage

I envision we can deprecate execution_order_group after this

From @Luay-Sol 's above comment, it seems like we can use both execution order group and depends_on together, no?

@nitrocode While it is true there's no issue with having both, I see merit points to merging the two. At the moment it is confusing to have 2 features that fundamentally try to do the same thing, i.e create an execution order, but each one handles a different use case, explicit project/workspace apply vs global apply.

The way I envision this is if we get the depend_on to do both, it will be much simpler for the user, to define it only once.

nitrocode · 2023-04-18T22:10:36Z

ah ok, ty for the clarification.

jamengual · 2023-04-20T04:12:10Z

For your second point, It is not complicated to add such a key. However, I would like to better understand the use case. I am trying to figure out when a softfail will be used. If it is just to prevent failure, maybe not define any dependencies to the project.

I think @GenPage is referring to this PR use case? #3323

GenPage · 2023-04-20T18:57:37Z

Thanks for your feedback @GenPage. For your first point about deprecating execution_order_group, I agree and I think in the next iteration we can beef up depends_on and have it handle both cases (explicit project/workspace apply & global apply).

@Luay-Sol Sounds good.

For your second point, It is not complicated to add such a key. However, I would like to better understand the use case. I am trying to figure out when a softfail will be used. If it is just to prevent failure, maybe not define any dependencies to the project.

The idea is you would still get ordering, but the project doesn't stop the continuation of dependency execution. If we were to deprecate execution_order_group that functionality that provides an ordered run without stopping would be gone.

e.g. "Hey, I want to execute in this order but I don't care if this project fails to apply, I want to rest of the dependencies to run"

We can address that in a separate PR, where-as this PR just adds depends_on

GenPage · 2023-05-06T04:14:34Z

@Luay-Sol Any update on addressing some of the code comments?

Luay-Sol · 2023-05-06T12:56:05Z

@Luay-Sol Any update on addressing some of the code comments?

@GenPage I will be getting back to this PR soon.

Luay-Sol · 2023-10-28T15:01:38Z

@jamengual @GenPage @vincentgna can I get final review on this? Anything else needed from me or I can do to get it out? Thanks.

GenPage

@Luay-Sol Excellent work, thank you for taking the time to contribute this.

GenPage · 2023-12-11T17:53:05Z

@Luay-Sol Are you able to resolve the merge conflicts locally? We cannot resolve them via the web UI. Can you rebase from the Atlantis main branch please?

glasser · 2023-12-12T17:06:54Z

Hi, just to check my understanding — in earlier discussions on this PR, the idea that this replaces execution_order_group is discussed. In the actual version merged, depends_on only adds potential validation errors and does not affect ordering when you run plan or apply without a specific subdirectory, right? So likely folks will want to use them in conjunction with each other?

Luay-Sol · 2023-12-13T01:48:28Z

Hi, just to check my understanding — in earlier discussions on this PR, the idea that this replaces execution_order_group is discussed. In the actual version merged, depends_on only adds potential validation errors and does not affect ordering when you run plan or apply without a specific subdirectory, right? So likely folks will want to use them in conjunction with each other?

@glasser yes, you understood it correctly. Here's an example of how I use it:

projects:
  - name: development
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/development.tfvars"]
    execution_order_group: 1
    workspace: development
    workflow: infra
  - name: rc
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/rc.tfvars"]
    depends_on: ["development"]
    execution_order_group: 2    
    workspace: stability-test
    workflow: infra
  - name: staging
    dir: .
    autoplan:
      when_modified: ["*.tf", "vars/staging.tfvars"]
    depends_on: ["rc"]
    execution_order_group: 3
    workspace: staging
    workflow: infra 
    ```

* feat: implemented the code for the depends on functionality Co-authored-by: Vincent <106497818+vincentgna@users.noreply.github.com>

Ref: - runatlantis#3292

Luay-Sol added 2 commits March 31, 2023 17:29

feat: implemented the code for the depends on functionnality

8b27e2e

feat: implemented the code for the depends on functionnality

ec74685

Luay-Sol requested a review from a team as a code owner April 4, 2023 14:23

Merge branch 'main' into add-project-depends-on-functionnality

5c50881

github-actions bot added docs Documentation go Pull requests that update Go code labels Apr 4, 2023

nitrocode mentioned this pull request Apr 4, 2023

Add company Logos and success histories to the Atlantis page #2626

Open

Luay-Sol mentioned this pull request Apr 4, 2023

Support prerequisite apply of projects for apply_requirements #391

Open

jamengual mentioned this pull request Apr 7, 2023

Feature: Priority Directories, Priority Plans #3241

Open

1 task

jamengual reviewed Apr 8, 2023

View reviewed changes

GenPage requested changes Apr 14, 2023

View reviewed changes

GenPage mentioned this pull request Apr 16, 2023

feat: when using order group, abort plan/apply if any fail #3323

Merged

2 tasks

GenPage added this to the v0.24.0 milestone May 6, 2023

GenPage added the waiting-on-response Waiting for a response from the user label May 6, 2023

GenPage added work-in-progress and removed waiting-on-response Waiting for a response from the user labels May 15, 2023

GenPage modified the milestones: v0.24.0, v0.25.0 May 15, 2023

chore: address PR comments

fda66b7

Luay-Sol requested a review from vincentgna October 16, 2023 15:10

Merge branch 'main' into add-project-depends-on-functionnality

89e4253

jamengual added waiting-on-review Waiting for a review from a maintainer and removed waiting-on-response Waiting for a response from the user labels Nov 15, 2023

jamengual modified the milestones: v0.25.0, v0.26.0 Nov 15, 2023

GenPage modified the milestones: v0.26.0, v0.27.0 Dec 11, 2023

GenPage previously approved these changes Dec 11, 2023

View reviewed changes

GenPage self-assigned this Dec 11, 2023

GenPage added waiting-on-response Waiting for a response from the user feature New functionality/enhancement and removed waiting-on-review Waiting for a review from a maintainer work-in-progress labels Dec 11, 2023

jamengual previously approved these changes Dec 11, 2023

View reviewed changes

Merge branch 'main' into add-project-depends-on-functionnality

3ea3bc9

Luay-Sol dismissed stale reviews from jamengual and GenPage via 3ea3bc9 December 11, 2023 20:14

jamengual approved these changes Dec 11, 2023

View reviewed changes

jamengual merged commit f79ece8 into runatlantis:main Dec 11, 2023
24 checks passed

BrewTestBot mentioned this pull request Dec 12, 2023

atlantis 0.27.0 Homebrew/homebrew-core#157029

Merged

glasser mentioned this pull request Dec 14, 2023

Support generating new depends_on key transcend-io/terragrunt-atlantis-config#296

Open

vincentgna added a commit to vincentgna/atlantis that referenced this pull request Feb 20, 2024

fix: Adopt depends_on bug fix

3880eb3

Ref: - runatlantis#3292

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add project depends on functionality #3292

feat: Add project depends on functionality #3292

Luay-Sol commented Apr 4, 2023

jamengual Apr 8, 2023

jamengual commented Apr 8, 2023

jamengual commented Apr 8, 2023

Luay-Sol commented Apr 14, 2023 •

edited

jamengual commented Apr 14, 2023

GenPage left a comment •

edited

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

GenPage Apr 14, 2023

vincentgna Oct 12, 2023 •

edited

vincentgna Oct 12, 2023 •

edited

GenPage Apr 14, 2023

nitrocode commented Apr 18, 2023

Luay-Sol commented Apr 18, 2023

Luay-Sol commented Apr 18, 2023

nitrocode commented Apr 18, 2023

jamengual commented Apr 20, 2023

GenPage commented Apr 20, 2023

GenPage commented May 6, 2023

Luay-Sol commented May 6, 2023

Luay-Sol commented Oct 28, 2023

GenPage left a comment

GenPage commented Dec 11, 2023

glasser commented Dec 12, 2023

Luay-Sol commented Dec 13, 2023

		@@ -121,6 +127,8 @@ func (p Project) ToValid() valid.Project {

		v.Name = p.Name

		v.Dependencies = p.Dependencies

feat: Add project depends on functionality #3292

feat: Add project depends on functionality #3292

Conversation

Luay-Sol commented Apr 4, 2023

what

why

tests

references

Choose a reason for hiding this comment

jamengual commented Apr 8, 2023

jamengual commented Apr 8, 2023

Luay-Sol commented Apr 14, 2023 • edited

jamengual commented Apr 14, 2023

GenPage left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentgna Oct 12, 2023 • edited

Choose a reason for hiding this comment

vincentgna Oct 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitrocode commented Apr 18, 2023

Luay-Sol commented Apr 18, 2023

Luay-Sol commented Apr 18, 2023

nitrocode commented Apr 18, 2023

jamengual commented Apr 20, 2023

GenPage commented Apr 20, 2023

GenPage commented May 6, 2023

Luay-Sol commented May 6, 2023

Luay-Sol commented Oct 28, 2023

GenPage left a comment

Choose a reason for hiding this comment

GenPage commented Dec 11, 2023

glasser commented Dec 12, 2023

Luay-Sol commented Dec 13, 2023

Luay-Sol commented Apr 14, 2023 •

edited

GenPage left a comment •

edited

vincentgna Oct 12, 2023 •

edited

vincentgna Oct 12, 2023 •

edited