From aa3729b64919d4e87f74ae1e0f4dcf2f8e567bb2 Mon Sep 17 00:00:00 2001 From: "docs-sourcer[bot]" <99042413+docs-sourcer[bot]@users.noreply.github.com> Date: Fri, 27 Jan 2023 14:02:32 +0000 Subject: [PATCH 1/3] Updated with the latest changes from the knowledge base discussions. --- docs/discussions/knowledge-base/647.mdx | 27 +++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 docs/discussions/knowledge-base/647.mdx diff --git a/docs/discussions/knowledge-base/647.mdx b/docs/discussions/knowledge-base/647.mdx new file mode 100644 index 0000000000..99c90e96bb --- /dev/null +++ b/docs/discussions/knowledge-base/647.mdx @@ -0,0 +1,27 @@ +--- +hide_table_of_contents: true +hide_title: true +custom_edit_url: null +--- + +import CenterLayout from "/src/components/CenterLayout" +import GitHub from "/src/components/GitHub" + + + + + + +Knowledge Base +

How do I operate on IAM resources when using aws-vault?

+ I am using `aws-vault` to manage access to my AWS accounts. When I attempt to run the `exec` command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using `aws-vault`?\r\n\r\n```bash\r\naws-vault exec -- aws iam list-users\r\n```\r\n---\r\n\r\n\r\n

Tracked in ticket #109838

\r\n\r\n","bodyHTML":"

A customer asked:

\n
\n

I am using aws-vault to manage access to my AWS accounts. When I attempt to run the exec command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using aws-vault?

\n
\n
aws-vault exec <account-profile>-- aws iam list-users
\n
\n\n

Tracked in ticket #109838

\n","answer":{"body":"The temporary session token returned by [the AWS STS action `GetSessionToken`](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) is valid for making API calls to AWS services - with the following important exception: \r\n\r\n> You cannot call any IAM API operations unless MFA authentication information is included in the request. \r\n\r\nThis causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the `--no-session` flag to their `aws-vault exec` commands.\r\n\r\n\r\nSee also: https://github.com/99designs/aws-vault/issues/1056","bodyHTML":"

The temporary session token returned by the AWS STS action GetSessionToken is valid for making API calls to AWS services - with the following important exception:

\n
\n

You cannot call any IAM API operations unless MFA authentication information is included in the request.

\n
\n

This causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the --no-session flag to their aws-vault exec commands.

\n

See also: 99designs/aws-vault#1056

"}}} /> + + + + + From a5adba574ec18bc0cb238cb1c093eb1b4c0ed22c Mon Sep 17 00:00:00 2001 From: "docs-sourcer[bot]" <99042413+docs-sourcer[bot]@users.noreply.github.com> Date: Fri, 27 Jan 2023 14:21:30 +0000 Subject: [PATCH 2/3] Updated with the latest changes from the knowledge base discussions. --- docs/discussions/knowledge-base/647.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/discussions/knowledge-base/647.mdx b/docs/discussions/knowledge-base/647.mdx index 99c90e96bb..9e0cfab5b3 100644 --- a/docs/discussions/knowledge-base/647.mdx +++ b/docs/discussions/knowledge-base/647.mdx @@ -14,7 +14,7 @@ import GitHub from "/src/components/GitHub" Knowledge Base

How do I operate on IAM resources when using aws-vault?

- I am using `aws-vault` to manage access to my AWS accounts. When I attempt to run the `exec` command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using `aws-vault`?\r\n\r\n```bash\r\naws-vault exec -- aws iam list-users\r\n```\r\n---\r\n\r\n\r\n

Tracked in ticket #109838

\r\n\r\n","bodyHTML":"

A customer asked:

\n
\n

I am using aws-vault to manage access to my AWS accounts. When I attempt to run the exec command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using aws-vault?

\n
\n
aws-vault exec <account-profile>-- aws iam list-users
\n
\n\n

Tracked in ticket #109838

\n","answer":{"body":"The temporary session token returned by [the AWS STS action `GetSessionToken`](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) is valid for making API calls to AWS services - with the following important exception: \r\n\r\n> You cannot call any IAM API operations unless MFA authentication information is included in the request. \r\n\r\nThis causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the `--no-session` flag to their `aws-vault exec` commands.\r\n\r\n\r\nSee also: https://github.com/99designs/aws-vault/issues/1056","bodyHTML":"

The temporary session token returned by the AWS STS action GetSessionToken is valid for making API calls to AWS services - with the following important exception:

\n
\n

You cannot call any IAM API operations unless MFA authentication information is included in the request.

\n
\n

This causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the --no-session flag to their aws-vault exec commands.

\n

See also: 99designs/aws-vault#1056

"}}} /> + I am using `aws-vault` to manage access to my AWS accounts. When I attempt to run the `exec` command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using `aws-vault`?\r\n\r\n```bash\r\naws-vault exec -- aws iam list-users\r\n```\r\n---\r\n\r\n\r\n

Tracked in ticket #109838

\r\n\r\n","bodyHTML":"

A customer asked:

\n
\n

I am using aws-vault to manage access to my AWS accounts. When I attempt to run the exec command to delete IAM resources, I get a security token error. What do I need to do to be able to delete IAM resources when using aws-vault?

\n
\n
aws-vault exec <account-profile>-- aws iam list-users
\n
\n\n

Tracked in ticket #109838

\n","answer":{"body":"\r\n![vaultsession](https://user-images.githubusercontent.com/1769996/215109078-8a999555-dc46-4049-8cb4-d7f20ba37820.gif)\r\nThe temporary session token returned by [the AWS STS action `GetSessionToken`](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) is valid for making API calls to AWS services - with the following important exception: \r\n\r\n> You cannot call any IAM API operations unless MFA authentication information is included in the request. \r\n\r\nThis causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the `--no-session` flag to their `aws-vault exec` commands.\r\n\r\n\r\nSee also: https://github.com/99designs/aws-vault/issues/1056","bodyHTML":"

\"vaultsession\"
\nThe temporary session token returned by the AWS STS action GetSessionToken is valid for making API calls to AWS services - with the following important exception:

\n
\n

You cannot call any IAM API operations unless MFA authentication information is included in the request.

\n
\n

This causes confusion to users of aws-vault who are using an IAM user in their profile. When they attempt to use calls for IAM they get InvalidClientTokenId, and ultimately are forced to expose their permanent credentials by passing the --no-session flag to their aws-vault exec commands.

\n

See also: 99designs/aws-vault#1056

"}}} /> @@ -22,6 +22,6 @@ import GitHub from "/src/components/GitHub" From 7fbef85376f49b02c64b3201b1959cea2ddc702f Mon Sep 17 00:00:00 2001 From: "docs-sourcer[bot]" <99042413+docs-sourcer[bot]@users.noreply.github.com> Date: Fri, 27 Jan 2023 16:38:13 +0000 Subject: [PATCH 3/3] Updated with the latest changes from the knowledge base discussions. --- docs/discussions/knowledge-base/463.mdx | 4 ++-- docs/discussions/knowledge-base/650.mdx | 27 +++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) create mode 100644 docs/discussions/knowledge-base/650.mdx diff --git a/docs/discussions/knowledge-base/463.mdx b/docs/discussions/knowledge-base/463.mdx index 658dc86f29..4818639536 100644 --- a/docs/discussions/knowledge-base/463.mdx +++ b/docs/discussions/knowledge-base/463.mdx @@ -14,7 +14,7 @@ import GitHub from "/src/components/GitHub" Knowledge Base

Multiple terraform sources

- \r\n }\r\n}\r\n\r\ninclude another {\r\n terraform {\r\n source = \r\n }\r\n}\r\n```\r\nAny guidance would be much appreciated.\r\n\r\nThanks.\n\n---\n\n\n

Tracked in ticket #108752

\n\n","bodyHTML":"

So i have my TG directory structure as below

\n
.\n├── env1\n│   └── applications\n│       ├── app1\n│       │   └── terragrunt.hcl\n│       ├── app2\n│       │   └── terragrunt.hcl\n│       └── app3\n│           └── terragrunt.hcl\n├── env2\n│   └── applications\n│       └── app1\n│           └── terragrunt.hcl\n├── env3\n├── main.hcl\n└── modules\n    ├── app1\n    │   ├── main1.tf\n    │   └── main2.tf\n    └── app2\n        └── main2.tf\n
\n

I have couple of environments env1, env2, env3. Each env has app1, app2... applications.
\nThe applications/app1, applications/app2 ... directories only contains terragrunt.hcl and a config.tfvars where i set variables relevant to that app.
\nThe terragrunt.hcl file is all same. It simply does a find_in_parent_folders('main.hcl').

\n

The main.hcl simply does a

\n
locals {\n  root_dir = get_parent_terragrunt_dir()\n  relative_path = path_relative_to_include()\n deployment_path_components = compact(split(\"/\", local.relative_path))\n\n app = reverse(local.deployment_path_components)[0]\n}\n\nterraform {\n    source = \"${local.root_dir}/..//modules/${local.app}\"\n}\n
\n

The whole idea is that when i execute terragrunt plan/apply from say app2, it just runs terraform tf files from modules/app2. The config.tfvars of app2 drives what variation of app2 it should provision for that environment.

\n

This works as expected.

\n

However, Now i have a need to also run terraform tf files from another module (another/main1.tf and another/main2.tf) when I run say app1. So when I run terragrunt plan/apply from say app1, it should copy/run whatever is in modules/app1 + modules/another.

\n

I have tried using dependency which works but i was trying to see if there was a way to perhaps merge the source attribute of terraform block to include both this paths or some other better way of doing it ? I want to keep modules/app1 and modules/another as separate modules and include them both via applications/app1/terragrunt.hcl.

\n
└── modules\n    ├── app1\n    │   ├── main1.tf\n    │   └── main2.tf\n    └── app2\n        └── main2.tf\n    └── another\n        └── main1.tf\n        └── main2.tf\n
\n

I also tried multiple includes within applications/app1/terragrunt.hcl like below. But i noticed that whatever is the second terraform source that gets run. The below would run tf files from modules/another and not from modules/app1

\n
include app {\n  terraform {\n    source = <path-to-modules/app1> \n  }\n}\n\ninclude another {\n  terraform {\n    source = <path-to-modules/another> \n  }\n}\n
\n

Any guidance would be much appreciated.

\n

Thanks.

\n
\n\n

Tracked in ticket #108752

\n","answer":{"body":"This is not a supported feature of Terragrunt. The feature request ticket for this is https://github.com/gruntwork-io/terragrunt/issues/1462, but as indicated [in this comment](https://github.com/gruntwork-io/terragrunt/issues/1462#issuecomment-742559624), supporting that operating model requires a major overhaul of Terragrunt internals.\r\n\r\nThe best workaround currently is to generate a Terraform module that calls those modules using `module` blocks on the fly using `generate` blocks.","bodyHTML":"

This is not a supported feature of Terragrunt. The feature request ticket for this is gruntwork-io/terragrunt#1462, but as indicated in this comment, supporting that operating model requires a major overhaul of Terragrunt internals.

\n

The best workaround currently is to generate a Terraform module that calls those modules using module blocks on the fly using generate blocks.

"}}} /> + \r\n }\r\n}\r\n\r\ninclude another {\r\n terraform {\r\n source = \r\n }\r\n}\r\n```\r\nAny guidance would be much appreciated.\r\n\r\nThanks.\n\n---\n\n\n

Tracked in ticket #108752

\n\n","bodyHTML":"

So i have my TG directory structure as below

\n
.\n├── env1\n│   └── applications\n│       ├── app1\n│       │   └── terragrunt.hcl\n│       ├── app2\n│       │   └── terragrunt.hcl\n│       └── app3\n│           └── terragrunt.hcl\n├── env2\n│   └── applications\n│       └── app1\n│           └── terragrunt.hcl\n├── env3\n├── main.hcl\n└── modules\n    ├── app1\n    │   ├── main1.tf\n    │   └── main2.tf\n    └── app2\n        └── main2.tf\n
\n

I have couple of environments env1, env2, env3. Each env has app1, app2... applications.
\nThe applications/app1, applications/app2 ... directories only contains terragrunt.hcl and a config.tfvars where i set variables relevant to that app.
\nThe terragrunt.hcl file is all same. It simply does a find_in_parent_folders('main.hcl').

\n

The main.hcl simply does a

\n
locals {\n  root_dir = get_parent_terragrunt_dir()\n  relative_path = path_relative_to_include()\n deployment_path_components = compact(split(\"/\", local.relative_path))\n\n app = reverse(local.deployment_path_components)[0]\n}\n\nterraform {\n    source = \"${local.root_dir}/..//modules/${local.app}\"\n}\n
\n

The whole idea is that when i execute terragrunt plan/apply from say app2, it just runs terraform tf files from modules/app2. The config.tfvars of app2 drives what variation of app2 it should provision for that environment.

\n

This works as expected.

\n

However, Now i have a need to also run terraform tf files from another module (another/main1.tf and another/main2.tf) when I run say app1. So when I run terragrunt plan/apply from say app1, it should copy/run whatever is in modules/app1 + modules/another.

\n

I have tried using dependency which works but i was trying to see if there was a way to perhaps merge the source attribute of terraform block to include both this paths or some other better way of doing it ? I want to keep modules/app1 and modules/another as separate modules and include them both via applications/app1/terragrunt.hcl.

\n
└── modules\n    ├── app1\n    │   ├── main1.tf\n    │   └── main2.tf\n    └── app2\n        └── main2.tf\n    └── another\n        └── main1.tf\n        └── main2.tf\n
\n

I also tried multiple includes within applications/app1/terragrunt.hcl like below. But i noticed that whatever is the second terraform source that gets run. The below would run tf files from modules/another and not from modules/app1

\n
include app {\n  terraform {\n    source = <path-to-modules/app1> \n  }\n}\n\ninclude another {\n  terraform {\n    source = <path-to-modules/another> \n  }\n}\n
\n

Any guidance would be much appreciated.

\n

Thanks.

\n
\n\n

Tracked in ticket #108752

\n","answer":{"body":"This is not a supported feature of Terragrunt. The feature request ticket for this is https://github.com/gruntwork-io/terragrunt/issues/1462, but as indicated [in this comment](https://github.com/gruntwork-io/terragrunt/issues/1462#issuecomment-742559624), supporting that operating model requires a major overhaul of Terragrunt internals.\r\n\r\nThe best workaround currently is to generate a Terraform module that calls those modules using `module` blocks on the fly using `generate` blocks.","bodyHTML":"

This is not a supported feature of Terragrunt. The feature request ticket for this is gruntwork-io/terragrunt#1462, but as indicated in this comment, supporting that operating model requires a major overhaul of Terragrunt internals.

\n

The best workaround currently is to generate a Terraform module that calls those modules using module blocks on the fly using generate blocks.

"}}} /> @@ -22,6 +22,6 @@ import GitHub from "/src/components/GitHub" diff --git a/docs/discussions/knowledge-base/650.mdx b/docs/discussions/knowledge-base/650.mdx new file mode 100644 index 0000000000..823119c1b3 --- /dev/null +++ b/docs/discussions/knowledge-base/650.mdx @@ -0,0 +1,27 @@ +--- +hide_table_of_contents: true +hide_title: true +custom_edit_url: null +--- + +import CenterLayout from "/src/components/CenterLayout" +import GitHub from "/src/components/GitHub" + + + + + + +Knowledge Base +

Bad GITHUB_OAUTH_TOKEN value error in Gruntwork CI/CD Pipelines or ECS Deploy Runner

+ Hello There! We're getting this error on the CI/CD workflow:\r\n> \r\n```bash\r\n2023-01-27 16:18:51 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.\r\n2023-01-27 16:18:51 [INFO] [gruntwork-install] Installing infrastructure-deployer...\r\n[fetch] time=\"2023-01-27T16:18:51Z\" level=error msg=\"\\nReceived an HTTP 401 Response when attempting to query the repo for its tags.\\n\\nThis means that either your GitHub oAuth Token is invalid, or that the token is valid but is being used to request access\\nto either a public repo or a private repo to which you don't have access.\\n\\nUnderlying error message:\\nReceived HTTP Response 401 while fetching releases for GitHub URL https://api.github.com/repos/gruntwork-io/terraform-aws-ci/tags?per_page=100. Full HTTP response: {\\\"message\\\":\\\"Bad credentials\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/rest\\\"}\\n\\n\"\r\nError: Process completed with exit code 1.\r\n```\r\n> Does anyone have any idea what might be going on?\r\n> Thanks!","bodyHTML":"

A customer asked:

\n
\n

Hello There! We're getting this error on the CI/CD workflow:

\n
\n
2023-01-27 16:18:51 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.\n2023-01-27 16:18:51 [INFO] [gruntwork-install] Installing infrastructure-deployer...\n[fetch] time=\"2023-01-27T16:18:51Z\" level=error msg=\"\\nReceived an HTTP 401 Response when attempting to query the repo for its tags.\\n\\nThis means that either your GitHub oAuth Token is invalid, or that the token is valid but is being used to request access\\nto either a public repo or a private repo to which you don't have access.\\n\\nUnderlying error message:\\nReceived HTTP Response 401 while fetching releases for GitHub URL https://api.github.com/repos/gruntwork-io/terraform-aws-ci/tags?per_page=100. Full HTTP response: {\\\"message\\\":\\\"Bad credentials\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/rest\\\"}\\n\\n\"\nError: Process completed with exit code 1.
\n
\n

Does anyone have any idea what might be going on?
\nThanks!

\n
","answer":{"body":"The following Gruntworks products use the same general pattern for securely accessing your GitHub repositories when performing a CI/CD job, without dangerously including your secrets in your Docker images or version control: \r\n\r\n* [ECS Deploy Runner](https://docs.gruntwork.io/reference/services/ci-cd-pipeline/ecs-deploy-runner)\r\n* [Pipelines](https://gruntwork.io/pipelines/)\r\n\r\n# General access pattern leveraging a GitHub Personal Access Token\r\n\r\n1. Your machine user's (CI/CD user's) GitHub Personal Access Token (PAT) is stored in AWS Secrets Manager within your Reference Architecture's shared account \r\n2. At runtime, ECS Deploy Runner / Pipelines fetches your GitHub PAT from Secrets Manager and makes it available to the running ECS task. In this way, we load your secret \"just in time\" and it only exists ephemerally within the volatile memory of the running ECS container. \r\n3. The GitHub PAT is used to perform `git` operations such as cloning your infrastructure-live repository into your running task's ECS container so that can be operated on by your pipeline\r\n\r\nTherefore, if you're encountering this error, it means that you need to ensure two different things about your token: \r\n1. [Ensure that your GitHub PAT is valid](https://github.com/gruntwork-io/knowledge-base/discussions/642)\r\n2. [Ensure that your GitHub PAT has the correct scopes attached to it](https://github.com/gruntwork-io/knowledge-base/discussions/643)\r\n\r\nPlease see the linked KB posts to each for guides and demos on how to quickly confirm if either situation applies to your token. ","bodyHTML":"

The following Gruntworks products use the same general pattern for securely accessing your GitHub repositories when performing a CI/CD job, without dangerously including your secrets in your Docker images or version control:

\n\n

General access pattern leveraging a GitHub Personal Access Token

\n
    \n
  1. Your machine user's (CI/CD user's) GitHub Personal Access Token (PAT) is stored in AWS Secrets Manager within your Reference Architecture's shared account
    2. \n
  2. At runtime, ECS Deploy Runner / Pipelines fetches your GitHub PAT from Secrets Manager and makes it available to the running ECS task. In this way, we load your secret \"just in time\" and it only exists ephemerally within the volatile memory of the running ECS container.
    3. \n
  3. The GitHub PAT is used to perform git operations such as cloning your infrastructure-live repository into your running task's ECS container so that can be operated on by your pipeline
    4. \n
\n

Therefore, if you're encountering this error, it means that you need to ensure two different things about your token:

\n
    \n
  1. Ensure that your GitHub PAT is valid
    2. \n
  2. Ensure that your GitHub PAT has the correct scopes attached to it
    3. \n
\n

Please see the linked KB posts to each for guides and demos on how to quickly confirm if either situation applies to your token.

"}}} /> + + + + +