gruntwork-io · eak12913 · May 26, 2023 · May 26, 2023
diff --git a/docs/discussions/knowledge-base/709.mdx b/docs/discussions/knowledge-base/709.mdx
@@ -0,0 +1,27 @@
+---
+hide_table_of_contents: true
+hide_title: true
+custom_edit_url: null
+---
+
+import CenterLayout from "/src/components/CenterLayout"
+import GitHub from "/src/components/GitHub"
+
+<head>
+  <link rel="canonical" href="https://github.com/gruntwork-io/knowledge-base/discussions/709" />
+</head>
+
+<CenterLayout>
+<span className="searchCategory">Knowledge Base</span>
+<h1>Terragrunt Github Pipeline with multiple environments and dependencies</h1>
+<GitHub discussion={{"id":"D_kwDOF8slf84ATwk2","number":709,"author":{"login":"lpossamai"},"title":"Terragrunt Github Pipeline with multiple environments and dependencies","body":"\r\nHi all :wave: \r\n\r\nI'm setting up a new Terragrunt repository and have few questions specially regarding the best practices around the pipeline. I'll be using Github Actions.\r\n\r\nMy current Terragrunt structure looks like this:\r\n```\r\n.\r\n├── config\r\n│   ├── _envcommon\r\n│   │   ├── README.md\r\n│   │   ├── s3.hcl\r\n│   │   └── s3-logging-bucket.hcl\r\n│   ├── Makefile\r\n├── environments\r\n│   ├── dev\r\n│   │   ├── ap-southeast-2\r\n│   │   │   ├── region.hcl\r\n│   │   │   ├── s3\r\n│   │   │   │   ├── Makefile\r\n│   │   │   │   └── terragrunt.hcl\r\n│   │   │   └── s3-logging-bucket\r\n│   │   │       ├── Makefile\r\n│   │   │       └── terragrunt.hcl\r\n│   │   ├── env.hcl\r\n│   │   └── Makefile\r\n│   ├── prod\r\n│   │   └── env.hcl\r\n│   └── staging\r\n│       └── env.hcl\r\n├── modules\r\n├── README.md\r\n└── terragrunt.hcl\r\n```\r\n\r\nThe **main** reason I'm using Terragrunt, is because after few years mainly using Terraform I ended up learning the hard way that Terraform sucks when there are multiple modules, state files and dependencies between them.\r\n\r\nLooks like [I cannot use `run-all` with Terragrunt](https://github.com/gruntwork-io/terragrunt/issues/720#issuecomment-497888756) when there are dependencies between modules. In that case, I'll have to make my CI to enter in each `environment/**/ap-southeast-2/**/` folder and run the commands there.\r\n\r\nHowever by doing that, I'll be creating lots and lots of duplicated files in the `.github/workflow` directory for each of the resources. Github Actions will need to perform the following:\r\n```\r\n[...]\r\ncd environment/dev/ap-southeast-2/s3\r\nterragrunt plan\r\n[...]\r\n\r\n[...]\r\ncd environment/dev/ap-southeast-2/s3-logging-bucket\r\nterragrunt plan\r\n[...]\r\n\r\n[...]\r\ncd environment/dev/ap-southeast-2/vpc\r\nterragrunt plan\r\n[...]\r\n\r\netc\r\n```\r\n\r\nWondering if there are better ways to achieve the same result? Any suggestions are more than welcome.\r\n\r\nCheers! :smile: \r\n\r\n---\r\n\r\n<ins datetime=\"2023-05-10T02:44:54Z\">\r\n  <p><a href=\"https://support.gruntwork.io/hc/requests/110155\">Tracked in ticket #110155</a></p>\r\n</ins>\r\n","bodyHTML":"<p dir=\"auto\">Hi all <g-emoji class=\"g-emoji\" alias=\"wave\" fallback-src=\"https://github.githubassets.com/images/icons/emoji/unicode/1f44b.png\">👋</g-emoji></p>\n<p dir=\"auto\">I'm setting up a new Terragrunt repository and have few questions specially regarding the best practices around the pipeline. I'll be using Github Actions.</p>\n<p dir=\"auto\">My current Terragrunt structure looks like this:</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\".\n├── config\n│   ├── _envcommon\n│   │   ├── README.md\n│   │   ├── s3.hcl\n│   │   └── s3-logging-bucket.hcl\n│   ├── Makefile\n├── environments\n│   ├── dev\n│   │   ├── ap-southeast-2\n│   │   │   ├── region.hcl\n│   │   │   ├── s3\n│   │   │   │   ├── Makefile\n│   │   │   │   └── terragrunt.hcl\n│   │   │   └── s3-logging-bucket\n│   │   │       ├── Makefile\n│   │   │       └── terragrunt.hcl\n│   │   ├── env.hcl\n│   │   └── Makefile\n│   ├── prod\n│   │   └── env.hcl\n│   └── staging\n│       └── env.hcl\n├── modules\n├── README.md\n└── terragrunt.hcl\"><pre class=\"notranslate\"><code class=\"notranslate\">.\n├── config\n│   ├── _envcommon\n│   │   ├── README.md\n│   │   ├── s3.hcl\n│   │   └── s3-logging-bucket.hcl\n│   ├── Makefile\n├── environments\n│   ├── dev\n│   │   ├── ap-southeast-2\n│   │   │   ├── region.hcl\n│   │   │   ├── s3\n│   │   │   │   ├── Makefile\n│   │   │   │   └── terragrunt.hcl\n│   │   │   └── s3-logging-bucket\n│   │   │       ├── Makefile\n│   │   │       └── terragrunt.hcl\n│   │   ├── env.hcl\n│   │   └── Makefile\n│   ├── prod\n│   │   └── env.hcl\n│   └── staging\n│       └── env.hcl\n├── modules\n├── README.md\n└── terragrunt.hcl\n</code></pre></div>\n<p dir=\"auto\">The <strong>main</strong> reason I'm using Terragrunt, is because after few years mainly using Terraform I ended up learning the hard way that Terraform sucks when there are multiple modules, state files and dependencies between them.</p>\n<p dir=\"auto\">Looks like <a href=\"https://github.com/gruntwork-io/terragrunt/issues/720#issuecomment-497888756\" data-hovercard-type=\"issue\" data-hovercard-url=\"/gruntwork-io/terragrunt/issues/720/hovercard\">I cannot use <code class=\"notranslate\">run-all</code> with Terragrunt</a> when there are dependencies between modules. In that case, I'll have to make my CI to enter in each <code class=\"notranslate\">environment/**/ap-southeast-2/**/</code> folder and run the commands there.</p>\n<p dir=\"auto\">However by doing that, I'll be creating lots and lots of duplicated files in the <code class=\"notranslate\">.github/workflow</code> directory for each of the resources. Github Actions will need to perform the following:</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"[...]\ncd environment/dev/ap-southeast-2/s3\nterragrunt plan\n[...]\n\n[...]\ncd environment/dev/ap-southeast-2/s3-logging-bucket\nterragrunt plan\n[...]\n\n[...]\ncd environment/dev/ap-southeast-2/vpc\nterragrunt plan\n[...]\n\netc\"><pre class=\"notranslate\"><code class=\"notranslate\">[...]\ncd environment/dev/ap-southeast-2/s3\nterragrunt plan\n[...]\n\n[...]\ncd environment/dev/ap-southeast-2/s3-logging-bucket\nterragrunt plan\n[...]\n\n[...]\ncd environment/dev/ap-southeast-2/vpc\nterragrunt plan\n[...]\n\netc\n</code></pre></div>\n<p dir=\"auto\">Wondering if there are better ways to achieve the same result? Any suggestions are more than welcome.</p>\n<p dir=\"auto\">Cheers! <g-emoji class=\"g-emoji\" alias=\"smile\" fallback-src=\"https://github.githubassets.com/images/icons/emoji/unicode/1f604.png\">😄</g-emoji></p>\n<hr>\n<ins datetime=\"2023-05-10T02:44:54Z\">\n  <p dir=\"auto\"><a href=\"https://support.gruntwork.io/hc/requests/110155\" rel=\"nofollow\">Tracked in ticket #110155</a></p>\n</ins>","answer":{"body":"> @lpossamai we have a similar issue when trying to run `terragrunt run-all init` and `terragrunt run-all plan` against all of our terragrunt directories. We're using terragrunt in a CICD pipeline via GitHub Actions and after a few iterations, we got around the output dependency issue by introducing the mock outputs, but now we get something like `terragrunt Underlying error: invalid character ':' after top-level value` when trying to do a `run-all plan`. Have you managed to find a workaround? In the interim, we've created a Makefile and are now running the int, plan and apply against the first directory then moving onto the second directory. But this is not ideal. Ideally, we want a separate task in GitHub Actions to run a `terragrunt run-all init` then a separate task to run a `terragrunt run-all plan` and then finally a separate task to run a `terragrunt run-all apply`. Any advice?\r\n\r\nHey! The only solution I found was to use [Atlantis](https://www.runatlantis.io/) or [Terrateam](https://terrateam.io/). I've been using Terrateam and the community there is great and really help you out! I strongly suggest you try them :) It's been working great for me!\r\n\r\nI even got in touch with [antonbabenko](https://github.com/antonbabenko) to ask for some advise but he also mentioned the two solutions above.\r\n\r\nGood luck!","bodyHTML":"<blockquote>\n<p dir=\"auto\"><a class=\"user-mention notranslate\" data-hovercard-type=\"user\" data-hovercard-url=\"/users/lpossamai/hovercard\" data-octo-click=\"hovercard-link-click\" data-octo-dimensions=\"link_type:self\" href=\"https://github.com/lpossamai\">@lpossamai</a> we have a similar issue when trying to run <code class=\"notranslate\">terragrunt run-all init</code> and <code class=\"notranslate\">terragrunt run-all plan</code> against all of our terragrunt directories. We're using terragrunt in a CICD pipeline via GitHub Actions and after a few iterations, we got around the output dependency issue by introducing the mock outputs, but now we get something like <code class=\"notranslate\">terragrunt Underlying error: invalid character ':' after top-level value</code> when trying to do a <code class=\"notranslate\">run-all plan</code>. Have you managed to find a workaround? In the interim, we've created a Makefile and are now running the int, plan and apply against the first directory then moving onto the second directory. But this is not ideal. Ideally, we want a separate task in GitHub Actions to run a <code class=\"notranslate\">terragrunt run-all init</code> then a separate task to run a <code class=\"notranslate\">terragrunt run-all plan</code> and then finally a separate task to run a <code class=\"notranslate\">terragrunt run-all apply</code>. Any advice?</p>\n</blockquote>\n<p dir=\"auto\">Hey! The only solution I found was to use <a href=\"https://www.runatlantis.io/\" rel=\"nofollow\">Atlantis</a> or <a href=\"https://terrateam.io/\" rel=\"nofollow\">Terrateam</a>. I've been using Terrateam and the community there is great and really help you out! I strongly suggest you try them :) It's been working great for me!</p>\n<p dir=\"auto\">I even got in touch with <a href=\"https://github.com/antonbabenko\">antonbabenko</a> to ask for some advise but he also mentioned the two solutions above.</p>\n<p dir=\"auto\">Good luck!</p>"}}} />
+
+</CenterLayout>
+
+
+<!-- ##DOCS-SOURCER-START
+{
+  "sourcePlugin": "github-discussions",
+  "hash": "b081e3fa899b5727e0191bacb1ebec3f"
+}
+##DOCS-SOURCER-END -->
diff --git a/docs/discussions/knowledge-base/97.mdx b/docs/discussions/knowledge-base/97.mdx
@@ -14,14 +14,14 @@ import GitHub from "/src/components/GitHub"
 <CenterLayout>
 <span className="searchCategory">Knowledge Base</span>
 <h1>Repo copier: 404 failed to parse unknown error format with Gitlab URL</h1>
-<GitHub discussion={{"id":"D_kwDOF8slf84AOQ48","number":97,"author":{"login":"zackproser"},"title":"Repo copier: 404 failed to parse unknown error format with Gitlab URL","body":"A customer asked: \r\n\r\n> I'm encountering an error where the repo-copier fails to parse a destination URL: \r\n\r\nThe example target is\r\n\r\n```\r\ngitlab https://domain.com/gitlab/project/repo\r\n```\r\n\r\nThe failure is \r\n```\r\nERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\r\n```\r\n\r\n","bodyHTML":"<p dir=\"auto\">A customer asked:</p>\n<blockquote>\n<p dir=\"auto\">I'm encountering an error where the repo-copier fails to parse a destination URL:</p>\n</blockquote>\n<p dir=\"auto\">The example target is</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"gitlab https://domain.com/gitlab/project/repo\"><pre class=\"notranslate\"><code class=\"notranslate\">gitlab https://domain.com/gitlab/project/repo\n</code></pre></div>\n<p dir=\"auto\">The failure is</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"ERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\"><pre class=\"notranslate\"><code class=\"notranslate\">ERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\n</code></pre></div>","answer":{"body":"The path should contain the full path to [a GitLab group](https://docs.gitlab.com/ee/user/group/). GitLab group is a kind of GitHub organization. Please make sure you specify the exact group, not a repository or user.\r\n\r\ncredit for this answer goes to @levkoburburas","bodyHTML":"<p dir=\"auto\">The path should contain the full path to <a href=\"https://docs.gitlab.com/ee/user/group/\" rel=\"nofollow\">a GitLab group</a>. GitLab group is a kind of GitHub organization. Please make sure you specify the exact group, not a repository or user.</p>\n<p dir=\"auto\">credit for this answer goes to <a class=\"user-mention notranslate\" data-hovercard-type=\"user\" data-hovercard-url=\"/users/levkoburburas/hovercard\" data-octo-click=\"hovercard-link-click\" data-octo-dimensions=\"link_type:self\" href=\"https://github.com/levkoburburas\">@levkoburburas</a></p>"}}} />
+<GitHub discussion={{"id":"D_kwDOF8slf84AOQ48","number":97,"author":{"login":"zackproser"},"title":"Repo copier: 404 failed to parse unknown error format with Gitlab URL","body":"A customer asked: \r\n\r\n> I'm encountering an error where the repo-copier fails to parse a destination URL: \r\n\r\nThe example target is\r\n\r\n```\r\ngitlab https://domain.com/gitlab/project/repo\r\n```\r\n\r\nThe failure is \r\n```\r\nERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\r\n```\r\n\r\n","bodyHTML":"<p dir=\"auto\">A customer asked:</p>\n<blockquote>\n<p dir=\"auto\">I'm encountering an error where the repo-copier fails to parse a destination URL:</p>\n</blockquote>\n<p dir=\"auto\">The example target is</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"gitlab https://domain.com/gitlab/project/repo\"><pre class=\"notranslate\"><code class=\"notranslate\">gitlab https://domain.com/gitlab/project/repo\n</code></pre></div>\n<p dir=\"auto\">The failure is</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"ERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\"><pre class=\"notranslate\"><code class=\"notranslate\">ERROR:  GET https://domain/api/v4/namespaces/gitlab/project/repo: 404 failed to parse unknown error format\n</code></pre></div>","answer":{"body":"The path should contain the full path to [a GitLab group](https://docs.gitlab.com/ee/user/group/). GitLab group is a kind of GitHub organization. Please make sure you specify the exact group, not a repository or user.\r\n\r\ncredit for this answer goes to @levkoburburas","bodyHTML":"<p dir=\"auto\">The path should contain the full path to <a href=\"https://docs.gitlab.com/ee/user/group/\" rel=\"nofollow\">a GitLab group</a>. GitLab group is a kind of GitHub organization. Please make sure you specify the exact group, not a repository or user.</p>\n<p dir=\"auto\">credit for this answer goes to @levkoburburas</p>"}}} />
 
 </CenterLayout>
 
 
 <!-- ##DOCS-SOURCER-START
 {
   "sourcePlugin": "github-discussions",
-  "hash": "2f7a76de6439e2866198c5c9b4487e1d"
+  "hash": "92e13c277eac1e5190e83c04f8d10e38"
 }
 ##DOCS-SOURCER-END -->