Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BuildTestCommands-[app]-[env] already exists in stack #1986

Closed
prisidio-z-bot opened this issue Feb 27, 2021 · 16 comments · Fixed by #2003
Closed

BuildTestCommands-[app]-[env] already exists in stack #1986

prisidio-z-bot opened this issue Feb 27, 2021 · 16 comments · Fixed by #2003
Labels
guidance Issue requesting guidance or information about usage

Comments

@prisidio-z-bot
Copy link

I have multiple services in my application. WRT to this issue when I used "test_commands" in more than one svc's pipeline file for the same application I get the "already exists in stack issue" when I run 'pipeline update'
In this case, I have 2 services (following the pattern of [app]-[env]-[svc])

  • [vault]-[dev]-[jasper-svc]
  • [vault]-[dev]-[jasper-e2e]

When I added test_commands to the first everything was fine. When I added to the second one and tried running 'pipeline update', I get the error.
@efekarakus
Copy link
Contributor

Hi @prisidio-z-bot !

Just to make sure I'm understanding the issue correctly:

  • You have two git repositories: jasper-svc and jasper-e2e
  • You ran pipeline init in both of these repositories generating two separate pipeline.yml files
  • You updated the test_commands field in both pipeline.yml files, but only one of them worked?

Am I understanding it correctly? If you don't mind would you mind copy/pasting your pipeline.yml files? that would help us debug the issue. Thanks!

@efekarakus efekarakus added the guidance Issue requesting guidance or information about usage label Mar 1, 2021
@rmarapp-dio
Copy link

rmarapp-dio commented Mar 1, 2021 •
edited by efekarakus
Loading

Thanks @efekarakus Yes, what you described above is correct.

pipeline.yml for jasper-svc

# This YAML file defines the relationship and deployment ordering of your environments.

# The name of the pipeline
name: pipeline-vault-prisidio-jasper-svc

# The version of the schema used in this template
version: 1

# This section defines the source artifacts.
source:
  # The name of the provider that is used to store the source artifacts.
  provider: GitHub
  # Additional properties that further specifies the exact location
  # the artifacts should be sourced from. For example, the GitHub provider
  # has the following properties: repository, branch.
  properties:
    access_token_secret: github-token-vault-jasper-svc
    branch: main
    repository: https://github.com/prisidio/jasper-svc

# The deployment section defines the order the pipeline will deploy
# to your environments.
stages:
  - # The name of the environment to deploy to.
    name: dev-local
    # Optional: flag for manual approval action before deployment.
    # requires_approval: true
    # Optional: use test commands to validate this stage of your build.
    # test_commands: [echo 'running tests', make test]

  - # The name of the environment to deploy to.
    name: dev
    test_commands:
      - echo 'running tests ...'
      - export COPILOT_SERVICE_DISCOVERY_ENDPOINT=vault.local
      - export COPILOT_APPLICATION_NAME=test
      - export COPILOT_ENVIRONMENT_NAME=dev
      - export USE_CONFIG_FILE=true
      - export COPILOT_SERVICE_NAME=jasper-svc
      - gradle test
    # Optional: flag for manual approval action before deployment.
    # requires_approval: true
    # Optional: use test commands to validate this stage of your build.
    # test_commands: [echo 'running tests', make test]

pipeline.yml fo jasper-e2e

# This YAML file defines the relationship and deployment ordering of your environments.

# The name of the pipeline
name: pipeline-vault-prisidio-jasper-e2e

# The version of the schema used in this template
version: 1

# This section defines the source artifacts.
source:
  # The name of the provider that is used to store the source artifacts.
  provider: GitHub
  # Additional properties that further specifies the exact location
  # the artifacts should be sourced from. For example, the GitHub provider
  # has the following properties: repository, branch.
  properties:
    access_token_secret: github-token-vault-jasper-e2e
    branch: main
    repository: https://github.com/prisidio/jasper-e2e

# The deployment section defines the order the pipeline will deploy
# to your environments.
stages:
  - # The name of the environment to deploy to.
    name: dev
    test_commands:
      - echo "running tests.."
      - gradle test

@prisidio-z-bot
Copy link
Author

prisidio-z-bot commented Mar 2, 2021 via email •
edited by efekarakus
Loading

@efekarakus
Copy link
Contributor

Hi @prisidio-z-bot @rmarapp-prisidio ! I saw some possibly sensitive values shared above. We'll investigate the issue today, but in the mean time, I'd recommend deleting the existing IAM user and creating a new one with the same permissions to be safe

@rmarapp-dio
Copy link

Sorry, I'm not following you.. Can you please elaborate? I'm not seeing any user information. My apologies.

@efekarakus
Copy link
Contributor

I edited the comments to remove them :), but if you look at the history you can see the values for AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID

@prisidio-z-bot
Copy link
Author

prisidio-z-bot commented Mar 2, 2021 via email

@huanjani
Copy link
Contributor

huanjani commented Mar 2, 2021 •
edited
Loading

Hello, @prisidio-z-bot @rmarapp-prisidio!

The second test commands are erroring out because the CodeBuild project names are the same. Unfortunately, we hadn't anticipated this particular use case-- at the moment, the test command name is made up of the app name and the env/stage name, and doesn't include the svc name.

As a workaround until we can change this, you can follow these steps to change the project name for the first set of test commands, so when the second one is generated it isn't a duplicate:

  1. Go into the AWS console and choose the CloudFormation service
  2. Find the pipeline-vault-prisidio-jasper-svc Stack
  3. Click the Update button, Edit in template designer, View in Designer
  4. In the template, you should see a portion that looks like this:
BuildTestCommandsdev:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: BuildTestCommands-vault-dev

Change the value in the Name field to something like BuildTestCommands-vault-dev-svc

  1. Click the Create Stack button (cloud w/ up arrow) and click through with Next until you Update Stack.

Now, when you run copilot pipeline update for your second pipeline with test commands, the stack name won't be a duplicate.

Please let us know how it goes!

@huanjani huanjani mentioned this issue Mar 2, 2021
Merged
@mergify mergify bot closed this as completed in #2003 Mar 3, 2021
mergify bot pushed a commit that referenced this issue Mar 3, 2021
@huanjani
fix(cicd): remove Name field for test command build projects (#2003)
d7a31fe 
This change gives Copilot pipelines more flexibility, as it enables users to build pipelines that share an app and env, but different repos/svcs or branches. We are seeing that customers want to build multiple pipelines for single apps, and this makes our pipeline functionality more extensible by avoiding overlapping CodeBuild project names. 

We have [255 characters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codebuild-project.html#cfn-codebuild-project-name) to work with, so stringing together `BuildTestCommands-[appName]-[env/stageName]-[repoName]-[branchName]` should be okay.

Fixes #1986.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
@huanjani
Copy link
Contributor

huanjani commented Mar 3, 2021

@prisidio-z-bot @rmarapp-prisidio:

Thanks for bringing this bug to our attention. A fix (#2003) will be included in the next release!

@rmarapp-dio
Copy link

rmarapp-dio commented Mar 3, 2021 via email

@huanjani
Copy link
Contributor

huanjani commented Mar 3, 2021

1.3 was our latest release, so it'll be in 1.4! 😄

@efekarakus
Copy link
Contributor

This is now released in v1.4.0! https://github.com/aws/copilot-cli/releases/tag/v1.4.0

@rmarapp-dio
Copy link

rmarapp-dio commented Mar 15, 2021 via email

@rmarapp-dio
Copy link

rmarapp-dio commented Mar 15, 2021 via email

@rmarapp-dio
Copy link

rmarapp-dio commented Mar 15, 2021 via email

@efekarakus
Copy link
Contributor

You'd need to re-run copilot pipeline update and that should unblock you 🙏

thrau pushed a commit to localstack/copilot-cli-local that referenced this issue Dec 9, 2022
@huanjani @thrau
fix(cicd): remove Name field for test command build projects (aws#2003)
da75297 
This change gives Copilot pipelines more flexibility, as it enables users to build pipelines that share an app and env, but different repos/svcs or branches. We are seeing that customers want to build multiple pipelines for single apps, and this makes our pipeline functionality more extensible by avoiding overlapping CodeBuild project names. 

We have [255 characters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codebuild-project.html#cfn-codebuild-project-name) to work with, so stringing together `BuildTestCommands-[appName]-[env/stageName]-[repoName]-[branchName]` should be okay.

Fixes aws#1986.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
guidance Issue requesting guidance or information about usage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(cicd): remove Name field for test command build projects huanjani/amazon-ecs-cli-v2
4 participants
@efekarakus @huanjani @rmarapp-dio @prisidio-z-bot