Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support set custom TF/OpenTofu binary. | If you use a custom Docker image build, please note that TERRAFORM_VERSION now must be provided #670

Merged
merged 36 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c85dfd5
Allowing setting custom TF binary
den-is May 4, 2024
30674bc
README update
den-is May 4, 2024
af193f1
Update README.md
den-is May 9, 2024
43edcea
Update README.md
den-is May 9, 2024
78f7f10
Update hooks/_common.sh
den-is May 9, 2024
45706a8
Update hooks/_common.sh
den-is May 9, 2024
4018878
Update hooks/_common.sh
den-is May 9, 2024
c25e0fb
Update hooks/_common.sh
den-is May 9, 2024
7c448a6
Update hooks/_common.sh
den-is May 9, 2024
b1dd3bb
Update hooks/_common.sh
den-is May 9, 2024
2e353ba
refactoring
den-is May 9, 2024
8ca6f2d
update terraform_providers_lock.sh
den-is May 9, 2024
d3c85b4
README update
den-is May 9, 2024
c73c2e8
add globals doc for common::get_tf_path
den-is May 9, 2024
5301302
centralize TF_PATH initialization
den-is May 9, 2024
53146fe
fix expected hook-config arg
den-is May 9, 2024
a6fedca
fix command checks
den-is May 9, 2024
2f6f7b0
README update
den-is May 11, 2024
200b96b
README update
den-is May 11, 2024
5091208
README update
den-is May 11, 2024
3f94d4f
README update
den-is May 11, 2024
741aac6
README update
den-is May 11, 2024
128d4fa
README update
den-is May 11, 2024
c20b5fd
Apply suggestions from code review
MaxymVlasov May 21, 2024
e7deaaf
Apply suggestions from code review
MaxymVlasov May 21, 2024
e771a4f
Update README.md
den-is May 21, 2024
258234f
README update
den-is May 21, 2024
3660e75
Discard changes to README.md
MaxymVlasov May 21, 2024
6670e51
Revert "Discard changes to README.md"
MaxymVlasov May 21, 2024
825b7f0
Revert "README update"
MaxymVlasov May 21, 2024
bc2766c
Switch from global env to local in-function constant
MaxymVlasov May 21, 2024
168dd6a
fixup. Useless suppresions
MaxymVlasov May 21, 2024
80d239d
Add tofu to Docker, make terraform and tofu non-required deps
MaxymVlasov May 21, 2024
c7ad7d1
Add tofu to ISSUE template
MaxymVlasov May 21, 2024
e3c66bd
Apply suggestions from code review
MaxymVlasov May 22, 2024
1cb551e
Extract only need binary file
MaxymVlasov May 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 10 additions & 5 deletions .github/.container-structure-test-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,6 @@ commandTests:
args: ["-V"]
expectedOutput: ["^pre-commit ([0-9]+\\.){2}[0-9]+\\n$"]

- name: "terraform"
command: "terraform"
args: ["-version"]
expectedOutput: ["^Terraform v([0-9]+\\.){2}[0-9]+\\n"]

- name: "gcc"
command: "gcc"
args: ["--version"]
Expand All @@ -30,6 +25,16 @@ commandTests:
args: ["--version"]
expectedOutput: ["^Infracost v([0-9]+\\.){2}[0-9]+"]

- name: "opentofu"
command: "tofu"
args: ["-version"]
expectedOutput: ["^OpenTofu v([0-9]+\\.){2}[0-9]+\\n"]

- name: "terraform"
command: "terraform"
args: ["-version"]
expectedOutput: ["^Terraform v([0-9]+\\.){2}[0-9]+\\n"]

- name: "terraform-docs"
command: "terraform-docs"
args: ["--version"]
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/bug_report_local_install.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ Linux DESKTOP-C7315EF 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43
bash << EOF
bash --version | head -n 1 2>/dev/null || echo "bash SKIPPED"
pre-commit --version 2>/dev/null || echo "pre-commit SKIPPED"
tofu --version | head -n 1 2>/dev/null || echo "opentofu SKIPPED"
terraform --version | head -n 1 2>/dev/null || echo "terraform SKIPPED"
python --version 2>/dev/null || echo "python SKIPPED"
python3 --version 2>/dev/null || echo "python3 SKIPPED"
Expand Down
20 changes: 13 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,18 @@ COPY tools/install/ /install/
# Install required tools
#
ARG PRE_COMMIT_VERSION=${PRE_COMMIT_VERSION:-latest}
ARG TERRAFORM_VERSION=${TERRAFORM_VERSION:-latest}

RUN touch /.env && \
if [ "$PRE_COMMIT_VERSION" = "false" ] || [ "$TERRAFORM_VERSION" = "false" ]; then \
if [ "$PRE_COMMIT_VERSION" = "false" ]; then \
echo "Vital software can't be skipped" && exit 1; \
fi


RUN /install/pre-commit.sh
RUN /install/terraform.sh

#
# Install tools
#
ARG OPENTOFU_VERSION=${OPENTOFU_VERSION:-false}
ARG TERRAFORM_VERSION=${TERRAFORM_VERSION:-false}

ARG CHECKOV_VERSION=${CHECKOV_VERSION:-false}
ARG HCLEDIT_VERSION=${HCLEDIT_VERSION:-false}
ARG INFRACOST_VERSION=${INFRACOST_VERSION:-false}
Expand All @@ -51,6 +49,9 @@ ARG TRIVY_VERSION=${TRIVY_VERSION:-false}
# specified in step below
ARG INSTALL_ALL=${INSTALL_ALL:-false}
RUN if [ "$INSTALL_ALL" != "false" ]; then \
echo "OPENTOFU_VERSION=latest" >> /.env && \
echo "TERRAFORM_VERSION=latest" >> /.env && \
\
echo "CHECKOV_VERSION=latest" >> /.env && \
echo "HCLEDIT_VERSION=latest" >> /.env && \
echo "INFRACOST_VERSION=latest" >> /.env && \
Expand All @@ -63,6 +64,9 @@ RUN if [ "$INSTALL_ALL" != "false" ]; then \
echo "TRIVY_VERSION=latest" >> /.env \
; fi

RUN /install/opentofu.sh
RUN /install/terraform.sh

RUN /install/checkov.sh
RUN /install/hcledit.sh
RUN /install/infracost.sh
Expand All @@ -79,7 +83,9 @@ RUN /install/trivy.sh
RUN . /.env && \
F=tools_versions_info && \
pre-commit --version >> $F && \
./terraform --version | head -n 1 >> $F && \
(if [ "$OPENTOFU_VERSION" != "false" ]; then echo "./tofu --version | head -n 1" >> $F; else echo "opentofu SKIPPED" >> $F ; fi) && \
(if [ "$TERRAFORM_VERSION" != "false" ]; then echo "./terraform --version | head -n 1" >> $F; else echo "terraform SKIPPED" >> $F ; fi) && \
\
(if [ "$CHECKOV_VERSION" != "false" ]; then echo "checkov $(checkov --version)" >> $F; else echo "checkov SKIPPED" >> $F ; fi) && \
(if [ "$HCLEDIT_VERSION" != "false" ]; then echo "hcledit $(./hcledit version)" >> $F; else echo "hcledit SKIPPED" >> $F ; fi) && \
(if [ "$INFRACOST_VERSION" != "false" ]; then echo "$(./infracost --version)" >> $F; else echo "infracost SKIPPED" >> $F ; fi) && \
Expand Down
46 changes: 31 additions & 15 deletions README.md
den-is marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ If you are using `pre-commit-terraform` already or want to support its developme
* [Table of content](#table-of-content)
* [How to install](#how-to-install)
* [1. Install dependencies](#1-install-dependencies)
* [1.1 Custom Terraform binaries and OpenTofu support](#11-custom-terraform-binaries-and-opentofu-support)
* [2. Install the pre-commit hook globally](#2-install-the-pre-commit-hook-globally)
* [3. Add configs and hooks](#3-add-configs-and-hooks)
* [4. Run](#4-run)
Expand Down Expand Up @@ -67,7 +68,7 @@ If you are using `pre-commit-terraform` already or want to support its developme
### 1. Install dependencies

* [`pre-commit`](https://pre-commit.com/#install),
<sub><sup>[`terraform`](https://www.terraform.io/downloads.html),
den-is marked this conversation as resolved.
Show resolved Hide resolved
<sub><sup>[`terraform`](https://www.terraform.io/downloads.html) or [`opentofu`](https://opentofu.org/docs/intro/install/),
<sub><sup>[`git`](https://git-scm.com/downloads),
<sub><sup>[BASH `3.2.57` or newer](https://www.gnu.org/software/bash/#download),
<sub><sup>Internet connection (on first run),
Expand All @@ -77,17 +78,31 @@ If you are using `pre-commit-terraform` already or want to support its developme
<sub><sup>Some basic physical laws,
<sub><sup>Hope that it all will work.
</sup></sub></sup></sub></sup></sub></sup></sub></sup></sub></sup></sub></sup></sub></sup></sub></sup></sub><br><br>
* [`checkov`](https://github.com/bridgecrewio/checkov) required for `terraform_checkov` hook.
* [`terraform-docs`](https://github.com/terraform-docs/terraform-docs) required for `terraform_docs` hook.
* [`terragrunt`](https://terragrunt.gruntwork.io/docs/getting-started/install/) required for `terragrunt_validate` hook.
* [`terrascan`](https://github.com/tenable/terrascan) required for `terrascan` hook.
* [`TFLint`](https://github.com/terraform-linters/tflint) required for `terraform_tflint` hook.
* [`TFSec`](https://github.com/liamg/tfsec) required for `terraform_tfsec` hook.
* [`Trivy`](https://github.com/aquasecurity/trivy) required for `terraform_trivy` hook.
* [`infracost`](https://github.com/infracost/infracost) required for `infracost_breakdown` hook.
* [`jq`](https://github.com/stedolan/jq) required for `terraform_validate` with `--retry-once-with-cleanup` flag, and for `infracost_breakdown` hook.
* [`tfupdate`](https://github.com/minamijoyo/tfupdate) required for `tfupdate` hook.
* [`hcledit`](https://github.com/minamijoyo/hcledit) required for `terraform_wrapper_module_for_each` hook.
* [`checkov`](https://github.com/bridgecrewio/checkov) required for `terraform_checkov` hook
* [`terraform-docs`](https://github.com/terraform-docs/terraform-docs) required for `terraform_docs` hook
* [`terragrunt`](https://terragrunt.gruntwork.io/docs/getting-started/install/) required for `terragrunt_validate` hook
* [`terrascan`](https://github.com/tenable/terrascan) required for `terrascan` hook
* [`TFLint`](https://github.com/terraform-linters/tflint) required for `terraform_tflint` hook
* [`TFSec`](https://github.com/liamg/tfsec) required for `terraform_tfsec` hook
* [`Trivy`](https://github.com/aquasecurity/trivy) required for `terraform_trivy` hook
* [`infracost`](https://github.com/infracost/infracost) required for `infracost_breakdown` hook
* [`jq`](https://github.com/stedolan/jq) required for `terraform_validate` with `--retry-once-with-cleanup` flag, and for `infracost_breakdown` hook
* [`tfupdate`](https://github.com/minamijoyo/tfupdate) required for `tfupdate` hook
* [`hcledit`](https://github.com/minamijoyo/hcledit) required for `terraform_wrapper_module_for_each` hook


#### 1.1 Custom Terraform binaries and OpenTofu support

It is possible to set custom path to `terraform` binary.
This makes it possible to use [OpenTofu](https://opentofu.org) binary `tofu` instead of `terraform`.

How binary discovery works and how you can redefine it (first matched takes precedence):

1. Check if per hook configuration `--hook-config=--tf-path=<path_to_binary_or_binary_name>` is set
2. Check if `PCT_TFPATH=<path_to_binary_or_binary_name>` environment variable is set
3. Check if `TERRAGRUNT_TFPATH=<path_to_binary_or_binary_name>` environment variable is set
4. Check if `terraform` binary can be found in the user's $PATH
5. Check if `tofu` binary can be found in the user's $PATH

<details><summary><b>Docker</b></summary><br>

Expand Down Expand Up @@ -120,17 +135,18 @@ To install a specific version of individual tools, define it using `--build-arg`
```bash
docker build -t pre-commit-terraform \
--build-arg PRE_COMMIT_VERSION=latest \
--build-arg TERRAFORM_VERSION=latest \
--build-arg OPENTOFU_VERSION=latest \
--build-arg TERRAFORM_VERSION=1.5.7 \
--build-arg CHECKOV_VERSION=2.0.405 \
--build-arg HCLEDIT_VERSION=latest \
--build-arg INFRACOST_VERSION=latest \
--build-arg TERRAFORM_DOCS_VERSION=0.15.0 \
--build-arg TERRAGRUNT_VERSION=latest \
--build-arg TERRASCAN_VERSION=1.10.0 \
--build-arg TFLINT_VERSION=0.31.0 \
--build-arg TFSEC_VERSION=latest \
--build-arg TRIVY_VERSION=latest \
--build-arg TFUPDATE_VERSION=latest \
--build-arg HCLEDIT_VERSION=latest \
--build-arg TRIVY_VERSION=latest \
.
```

Expand Down
62 changes: 59 additions & 3 deletions hooks/_common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,8 @@ function common::per_dir_hook {
# despite there's only one positional ARG left
local -a -r files=("$@")

local -r tf_path=$(common::get_tf_binary_path)

# check is (optional) function defined
if [ "$(type -t run_hook_on_whole_repo)" == function ] &&
# check is hook run via `pre-commit run --all`
Expand Down Expand Up @@ -383,7 +385,7 @@ function common::per_dir_hook {
pushd "$dir_path" > /dev/null
fi

per_dir_hook_unique_part "$dir_path" "$change_dir_in_unique_part" "$parallelism_disabled" "${args[@]}"
per_dir_hook_unique_part "$dir_path" "$change_dir_in_unique_part" "$parallelism_disabled" "$tf_path" "${args[@]}"
} &
pids+=("$!")

Expand Down Expand Up @@ -445,13 +447,66 @@ function common::colorify {
echo -e "${COLOR}${TEXT}${RESET}" >&2
}

#######################################################################
# Get Terraform/OpenTofu binary path
# Allows user to set the path to custom Terraform or OpenTofu binary
# Globals (init and populate):
# HOOK_CONFIG (array) arguments that configure hook behavior
# PCT_TFPATH (string) user defined env var with path to Terraform/OpenTofu binary
# TERRAGRUNT_TFPATH (string) user defined env var with path to Terraform/OpenTofu binary
# Outputs:
# If failed - exit 1 with error message about missing Terraform/OpenTofu binary
#######################################################################
function common::get_tf_binary_path {
local hook_config_tf_path

for config in "${HOOK_CONFIG[@]}"; do
if [[ $config == --tf-path=* ]]; then
hook_config_tf_path=${config#*=}
hook_config_tf_path=${hook_config_tf_path%;}
den-is marked this conversation as resolved.
Show resolved Hide resolved
break
fi
done

# direct hook config, has the highest precedence
if [[ $hook_config_tf_path ]]; then
echo "$hook_config_tf_path"
return

# environment variable
elif [[ $PCT_TFPATH ]]; then
echo "$PCT_TFPATH"
return

# Maybe there is a similar setting for Terragrunt already
elif [[ $TERRAGRUNT_TFPATH ]]; then
echo "$TERRAGRUNT_TFPATH"
return

# check if Terraform binary is available
elif command -v terraform &> /dev/null; then
command -v terraform
return

# finally, check if Tofu binary is available
elif command -v tofu &> /dev/null; then
command -v tofu
return

else
common::colorify "red" "Neither Terraform nor OpenTofu binary could be found. Please either set the \"--tf-path\" hook configuration argument, or set the \"PCT_TFPATH\" environment variable, or set the \"TERRAGRUNT_TFPATH\" environment variable, or install Terraform or OpenTofu globally."
exit 1
yermulnik marked this conversation as resolved.
Show resolved Hide resolved
fi
}

#######################################################################
# Run terraform init command
# Arguments:
# command_name (string) command that will tun after successful init
# dir_path (string) PATH to dir relative to git repo root.
# Can be used in error logging
# parallelism_disabled (bool) if true - skip lock mechanism
# tf_path (string) PATH to Terraform/OpenTofu binary
# Globals (init and populate):
# TF_INIT_ARGS (array) arguments for `terraform init` command
# TF_PLUGIN_CACHE_DIR (string) user defined env var with name of the directory
Expand All @@ -464,6 +519,7 @@ function common::terraform_init {
local -r command_name=$1
local -r dir_path=$2
local -r parallelism_disabled=$3
local -r tf_path=$4

local exit_code=0
local init_output
Expand All @@ -480,13 +536,13 @@ function common::terraform_init {
# Plugin cache dir can't be written concurrently or read during write
# https://github.com/hashicorp/terraform/issues/31964
if [[ -z $TF_PLUGIN_CACHE_DIR || $parallelism_disabled == true ]]; then
init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
init_output=$($tf_path init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
exit_code=$?
else
# Locking just doesn't work, and the below works quicker instead. Details:
# https://github.com/hashicorp/terraform/issues/31964#issuecomment-1939869453
for i in {1..10}; do
init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
init_output=$($tf_path init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
exit_code=$?

if [ $exit_code -eq 0 ]; then
Expand Down
5 changes: 4 additions & 1 deletion hooks/terraform_checkov.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ function main {
# Availability depends on hook.
# parallelism_disabled (bool) if true - skip lock mechanism
# args (array) arguments that configure wrapped tool behavior
# tf_path (string) PATH to Terraform/OpenTofu binary
# Outputs:
# If failed - print out hook checks status
#######################################################################
Expand All @@ -46,7 +47,9 @@ function per_dir_hook_unique_part {
local -r change_dir_in_unique_part="$2"
# shellcheck disable=SC2034 # Unused var.
local -r parallelism_disabled="$3"
shift 3
# shellcheck disable=SC2034 # Unused var.
local -r tf_path="$4"
shift 4
local -a -r args=("$@")

checkov -d . "${args[@]}"
Expand Down
6 changes: 4 additions & 2 deletions hooks/terraform_fmt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ function main {
# Availability depends on hook.
# parallelism_disabled (bool) if true - skip lock mechanism
# args (array) arguments that configure wrapped tool behavior
# tf_path (string) PATH to Terraform/OpenTofu binary
# Outputs:
# If failed - print out hook checks status
#######################################################################
Expand All @@ -43,11 +44,12 @@ function per_dir_hook_unique_part {
local -r change_dir_in_unique_part="$2"
# shellcheck disable=SC2034 # Unused var.
local -r parallelism_disabled="$3"
shift 3
local -r tf_path="$4"
shift 4
local -a -r args=("$@")

# pass the arguments to hook
terraform fmt "${args[@]}"
$tf_path fmt "${args[@]}"

# return exit code to common::per_dir_hook
local exit_code=$?
Expand Down
8 changes: 5 additions & 3 deletions hooks/terraform_providers_lock.sh
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ function lockfile_contains_all_needed_sha {
# Availability depends on hook.
# parallelism_disabled (bool) if true - skip lock mechanism
# args (array) arguments that configure wrapped tool behavior
# tf_path (string) PATH to Terraform/OpenTofu binary
# Outputs:
# If failed - print out hook checks status
#######################################################################
Expand All @@ -95,7 +96,8 @@ function per_dir_hook_unique_part {
# shellcheck disable=SC2034 # Unused var.
local -r change_dir_in_unique_part="$2"
local -r parallelism_disabled="$3"
shift 3
local -r tf_path="$4"
shift 4
local -a -r args=("$@")

local platforms_count=0
Expand Down Expand Up @@ -138,7 +140,7 @@ function per_dir_hook_unique_part {
common::colorify "yellow" "DEPRECATION NOTICE: We introduced '--mode' flag for this hook.
Check migration instructions at https://github.com/antonbabenko/pre-commit-terraform#terraform_providers_lock
"
common::terraform_init 'terraform providers lock' "$dir_path" "$parallelism_disabled" || {
common::terraform_init 'terraform providers lock' "$dir_path" "$parallelism_disabled" "$tf_path" || {
exit_code=$?
return $exit_code
}
Expand All @@ -153,7 +155,7 @@ Check migration instructions at https://github.com/antonbabenko/pre-commit-terra
#? Don't require `tf init` for providers, but required `tf init` for modules
#? Mitigated by `function match_validate_errors` from terraform_validate hook
# pass the arguments to hook
terraform providers lock "${args[@]}"
$tf_path providers lock "${args[@]}"

# return exit code to common::per_dir_hook
exit_code=$?
Expand Down
5 changes: 4 additions & 1 deletion hooks/terraform_tflint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ function main {
# Availability depends on hook.
# parallelism_disabled (bool) if true - skip lock mechanism
# args (array) arguments that configure wrapped tool behavior
# tf_path (string) PATH to Terraform/OpenTofu binary
# Outputs:
# If failed - print out hook checks status
#######################################################################
Expand All @@ -54,7 +55,9 @@ function per_dir_hook_unique_part {
local -r change_dir_in_unique_part="$2"
# shellcheck disable=SC2034 # Unused var.
local -r parallelism_disabled="$3"
shift 3
# shellcheck disable=SC2034 # Unused var.
local -r tf_path="$4"
shift 4
local -a -r args=("$@")

if [ "$change_dir_in_unique_part" == "delegate_chdir" ]; then
Expand Down
5 changes: 4 additions & 1 deletion hooks/terraform_tfsec.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ function main {
# Availability depends on hook.
# parallelism_disabled (bool) if true - skip lock mechanism
# args (array) arguments that configure wrapped tool behavior
# tf_path (string) PATH to Terraform/OpenTofu binary
# Outputs:
# If failed - print out hook checks status
#######################################################################
Expand All @@ -49,7 +50,9 @@ function per_dir_hook_unique_part {
local -r change_dir_in_unique_part="$2"
# shellcheck disable=SC2034 # Unused var.
local -r parallelism_disabled="$3"
shift 3
# shellcheck disable=SC2034 # Unused var.
local -r tf_path="$4"
shift 4
local -a -r args=("$@")

# pass the arguments to hook
Expand Down