From 6c6eca463a74fa2608cb3de3e03873765d46252f Mon Sep 17 00:00:00 2001 From: Maksym Vlasov Date: Sat, 17 Feb 2024 22:41:40 +0200 Subject: [PATCH] feat: Add parallelism to major chunk of hooks. Check `Parallelism` section in README (#620) ### Reasoning We have a GH workflow that runs lockflies updates every week (implementation and reasoning [here](https://grem1.in/post/terraform-lockfiles-maxymvlasov/)). It usually takes from 2h 30min to 3h 15min. That was fine for us, till we found that our GH runners, based on AWS EC2s, started silently failing after 30min "without recent logs", and that was fixed by crutch which sends a dummy log every 10min. However, during the debugging, I spent some time describing why hooks were not utilizing all the provided resources. And that means a waste of time and money, not only for that corner case but for every huge commit, which can cause opting out by `git commit -n` of using hooks locally for changes that affect many directories. ### Description of your changes * Add per-hook `--parallelism-limit` setting to `--hook-config`. Defaults to `number of logical CPUs - 1` * As quick tests show, ~5% of stacks face race condition problem, no matter if any locking mechanism exists or dirs try to init in parallel. I suppose the lock failed as it uses disk when hooks run in memory, so the creation of the lock can take some time as there bunch of caches between Mem and Disk. These milliseconds are enough to allow running a few `t init` in parallel. * Final implementation uses a retry mechanism for cases when race condition failed to `t init` directory. In quick tests, I can say that on big changes: * Up to 2000% speed increase for `terraform_validate`, and up to 500% - for other affected hooks. * When `--parallelism-limit=1` I observed an insignificant increase in time (about 5-10%) compared to v1.86.0 which has no parallelism at all. This may be the cost of maintaining parallelism or the result of external factors since the tests were not conducted in a vacuum. For small changes, improvements are less significant. ----- Other significant findings/solutions included to this PR: * feat: Investigate and fix issue with wrong CPU count for containers (#623) So, I found that `nproc` always shows how many CPUs available is. K8s "limits" and docker `--cpus` are throttling mechanisms, which do not hide the visibility of all cores. There are a few workarounds, but IMO, it is better to implement checks for that than do them >Workaround for docker - set `--cpuset-cpus` >Workaraund for K8s - somehow deal with [kubelet static CPU management policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#cpu-management-policies), as [recommend in Reddit](https://news.ycombinator.com/item?id=25224714) * Send all "colorify" logs through stderr, as make able to add user-facing-logs in functions that also need to return same value to the function-caller. Needed for `common::get_cpu_num` err_msg show up ------ * Count --parallelism-ci-cpu-cores only in edge-cases Details: https://github.com/antonbabenko/pre-commit-terraform/pull/620#pullrequestreview-1883528728 --------- Co-authored-by: George L. Yermulnik --- .../bug_report_local_install.md | 13 +- Dockerfile | 1 + README.md | 65 ++++++ hooks/_common.sh | 195 ++++++++++++++++-- hooks/terraform_checkov.sh | 5 +- hooks/terraform_fmt.sh | 5 +- hooks/terraform_providers_lock.sh | 6 +- hooks/terraform_tflint.sh | 5 +- hooks/terraform_tfsec.sh | 5 +- hooks/terraform_trivy.sh | 5 +- hooks/terraform_validate.sh | 8 +- hooks/terragrunt_fmt.sh | 5 +- hooks/terragrunt_validate.sh | 5 +- hooks/terrascan.sh | 5 +- hooks/tfupdate.sh | 5 +- 15 files changed, 292 insertions(+), 41 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report_local_install.md b/.github/ISSUE_TEMPLATE/bug_report_local_install.md index 544dc03ed..bfccad8ad 100644 --- a/.github/ISSUE_TEMPLATE/bug_report_local_install.md +++ b/.github/ISSUE_TEMPLATE/bug_report_local_install.md @@ -81,16 +81,17 @@ pre-commit --version 2>/dev/null || echo "pre-commit SKIPPE terraform --version | head -n 1 2>/dev/null || echo "terraform SKIPPED" python --version 2>/dev/null || echo "python SKIPPED" python3 --version 2>/dev/null || echo "python3 SKIPPED" -echo -n "checkov " && checkov --version 2>/dev/null || echo "checkov SKIPPED" +echo -n "checkov " && checkov --version 2>/dev/null || echo "SKIPPED" infracost --version 2>/dev/null || echo "infracost SKIPPED" terraform-docs --version 2>/dev/null || echo "terraform-docs SKIPPED" terragrunt --version 2>/dev/null || echo "terragrunt SKIPPED" -echo -n "terrascan " && terrascan version 2>/dev/null || echo "terrascan SKIPPED" +echo -n "terrascan " && terrascan version 2>/dev/null || echo "SKIPPED" tflint --version 2>/dev/null || echo "tflint SKIPPED" -echo -n "tfsec " && tfsec --version 2>/dev/null || echo "tfsec SKIPPED" -echo -n "trivy " && trivy --version 2>/dev/null || echo "tfsec SKIPPED" -echo -n "tfupdate " && tfupdate --version 2>/dev/null || echo "tfupdate SKIPPED" -echo -n "hcledit " && hcledit version 2>/dev/null || echo "hcledit SKIPPED" +echo -n "tfsec " && tfsec --version 2>/dev/null || echo "SKIPPED" +echo -n "trivy " && trivy --version 2>/dev/null || echo "SKIPPED" +echo -n "tfupdate " && tfupdate --version 2>/dev/null || echo "SKIPPED" +echo -n "hcledit " && hcledit version 2>/dev/null || echo "SKIPPED" +echo -n "flock " && flock --version 2>/dev/null || echo "SKIPPED" EOF --> diff --git a/Dockerfile b/Dockerfile index 83d0bc05b..7809f9f9e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -173,6 +173,7 @@ RUN . /.env && \ RUN . /.env && \ F=tools_versions_info && \ pre-commit --version >> $F && \ + echo "flock $(flock 2>&1 | head -n 1)" >> $F && \ ./terraform --version | head -n 1 >> $F && \ (if [ "$CHECKOV_VERSION" != "false" ]; then echo "checkov $(checkov --version)" >> $F; else echo "checkov SKIPPED" >> $F ; fi) && \ (if [ "$INFRACOST_VERSION" != "false" ]; then echo "$(./infracost --version)" >> $F; else echo "infracost SKIPPED" >> $F ; fi) && \ diff --git a/README.md b/README.md index dcafd39e1..df8c4d0b5 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ If you are using `pre-commit-terraform` already or want to support its developme * [All hooks: Usage of environment variables in `--args`](#all-hooks-usage-of-environment-variables-in---args) * [All hooks: Set env vars inside hook at runtime](#all-hooks-set-env-vars-inside-hook-at-runtime) * [All hooks: Disable color output](#all-hooks-disable-color-output) + * [Many hooks: Parallelism](#many-hooks-parallelism) * [checkov (deprecated) and terraform\_checkov](#checkov-deprecated-and-terraform_checkov) * [infracost\_breakdown](#infracost_breakdown) * [terraform\_docs](#terraform_docs) @@ -338,6 +339,70 @@ To disable color output for all hooks, set `PRE_COMMIT_COLOR=never` var. Eg: PRE_COMMIT_COLOR=never pre-commit run ``` +### Many hooks: Parallelism + +> All, except deprecated hooks: `checkov`, `terraform_docs_replace` and hooks which can't be paralleled this way: `infracost_breakdown`, `terraform_wrapper_module_for_each`. +> Also, there's a chance that parallelism have no effect on `terragrunt_fmt` and `terragrunt_validate` hooks + +By default, parallelism is set to `number of logical CPUs - 1`. +If you'd like to disable parallelism, set it to `1` + +```yaml +- id: terragrunt_validate + args: + - --hook-config=--parallelism-limit=1 +``` + +In the same way you can set it to any positive integer. + +If you'd like to set parallelism value relative to number of CPU logical cores - provide valid Bash arithmetic expression and use `CPU` as a reference to the number of CPU logical cores + + +```yaml +- id: terraform_providers_lock + args: + - --hook-config=--parallelism-limit=CPU*4 +``` + +> [!TIP] +>
Info useful for parallelism fine-tunning +> +>
+> Tests below were run on repo with 45 Terraform dirs on laptop with 16 CPUs, SSD and 1Gbit/s network. Laptop was slightly used in the process. +> +> Observed results may vary greatly depending on your repo structure, machine characteristics and their usage. +> +> If during fine-tuning you'll find that your results are very different from provided below and you think that this data could help someone else - feel free to send PR. +> +> +> | Hook | Most used resource | Comparison of optimization results / Notes | +> | ------------------------------------------------------------------------------ | ---------------------------------- | --------------------------------------------------------------- | +> | terraform_checkov | CPU heavy | - | +> | terraform_fmt | CPU heavy | - | +> | terraform_providers_lock (3 platforms,
`--mode=always-regenerate-lockfile`) | Network & Disk heavy | `defaults (CPU-1)` - 3m 39s; `CPU*2` - 3m 19s; `CPU*4` - 2m 56s | +> | terraform_tflint | CPU heavy | - | +> | terraform_tfsec | CPU heavy | - | +> | terraform_trivy | CPU moderate | `defaults (CPU-1)` - 32s; `CPU*2` - 30s; `CPU*4` - 31s | +> | terraform_validate (t validate only) | CPU heavy | - | +> | terraform_validate (t init + t validate) | Network & Disk heavy, CPU moderate | `defaults (CPU-1)` - 1m 30s; `CPU*2` - 1m 25s; `CPU*4` - 1m 41s | +> | terragrunt_fmt | CPU heavy | N/A? need more info from TG users | +> | terragrunt_validate | CPU heavy | N/A? need more info from TG users | +> | terrascan | CPU moderate-heavy | `defaults (CPU-1)` - 8s; `CPU*2` - 6s | +> | tfupdate | Disk/Network? | too quick in any settings. More info needed | +> +> +>
+ + + +```yaml +args: + - --hook-config=--parallelism-ci-cpu-cores=N +``` + +If you don't see code above in your `pre-commit-config.yaml` or logs - you don't need it. +`--parallelism-ci-cpu-cores` used only in edge cases and is ignored in other situations. Check out its usage in [hooks/_common.sh](hooks/_common.sh) + ### checkov (deprecated) and terraform_checkov > `checkov` hook is deprecated, please use `terraform_checkov`. diff --git a/hooks/_common.sh b/hooks/_common.sh index 31be98100..730848997 100644 --- a/hooks/_common.sh +++ b/hooks/_common.sh @@ -170,6 +170,75 @@ function common::is_hook_run_on_whole_repo { fi } +####################################################################### +# Get the number of CPU logical cores available for pre-commit to use +# Arguments: +# parallelism_ci_cpu_cores (string) Used in edge cases when number of +# CPU cores can't be derived automatically +# Outputs: +# Returns number of CPU logical cores, rounded down to nearest integer +####################################################################### +function common::get_cpu_num { + local -r parallelism_ci_cpu_cores=$1 + + local millicpu + + if [[ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]]; then + # Inside K8s pod or DinD in K8s + millicpu=$(< /sys/fs/cgroup/cpu/cpu.cfs_quota_us) + + if [[ $millicpu -eq -1 ]]; then + # K8s no limits or in DinD + if [[ -n $parallelism_ci_cpu_cores ]]; then + if [[ ! $parallelism_ci_cpu_cores =~ ^[[:digit:]]+$ ]]; then + common::colorify "yellow" "--parallelism-ci-cpu-cores set to" \ + "'$parallelism_ci_cpu_cores' which is not a positive integer.\n" \ + "To avoid possible harm, parallelism is disabled.\n" \ + "To re-enable it, change corresponding value in config to positive integer" + + echo 1 + return + fi + + echo "$parallelism_ci_cpu_cores" + return + fi + + common::colorify "yellow" "Unable to derive number of available CPU cores.\n" \ + "Running inside K8s pod without limits or inside DinD without limits propagation.\n" \ + "To avoid possible harm, parallelism is disabled.\n" \ + "To re-enable it, set corresponding limits, or set the following for the current hook:\n" \ + " args:\n" \ + " - --hook-config=--parallelism-ci-cpu-cores=N\n" \ + "where N is the number of CPU cores to allocate to pre-commit." + + echo 1 + return + fi + + echo $((millicpu / 1000)) + return + fi + + if [[ -f /sys/fs/cgroup/cpu.max ]]; then + # Inside Linux (Docker?) container + millicpu=$(cut -d' ' -f1 /sys/fs/cgroup/cpu.max) + + if [[ $millicpu == max ]]; then + # No limits + nproc 2> /dev/null || echo 1 + return + fi + + echo $((millicpu / 1000)) + return + fi + + # On host machine or any other case + # `nproc` - Linux/FreeBSD, `sysctl -n hw.ncpu` - macOS/BSD, `echo 1` - fallback + nproc 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1 +} + ####################################################################### # Hook execution boilerplate logic which is common to hooks, that run # on per dir basis. @@ -219,10 +288,16 @@ function common::per_dir_hook { # Lookup hook-config for modifiers that impact common behavior local change_dir_in_unique_part=false + + local parallelism_limit IFS=";" read -r -a configs <<< "${HOOK_CONFIG[*]}" for c in "${configs[@]}"; do IFS="=" read -r -a config <<< "$c" - key=${config[0]} + + # $hook_config receives string like '--foo=bar; --baz=4;' etc. + # It gets split by `;` into array, which we're parsing here ('--foo=bar' ' --baz=4') + # Next line removes leading spaces, to support >1 `--hook-config` args + key="${config[0]## }" value=${config[1]} case $key in @@ -233,32 +308,82 @@ function common::per_dir_hook { change_dir_in_unique_part="delegate_chdir" fi ;; + --parallelism-limit) + # this flag will limit the number of parallel processes + parallelism_limit="$value" + ;; + --parallelism-ci-cpu-cores) + # Used in edge cases when number of CPU cores can't be derived automatically + parallelism_ci_cpu_cores="$value" + ;; esac done + CPU=$(common::get_cpu_num "$parallelism_ci_cpu_cores") + # parallelism_limit can include reference to 'CPU' variable + local parallelism_disabled=false + + if [[ ! $parallelism_limit ]]; then + # Could evaluate to 0 + parallelism_limit=$((CPU - 1)) + elif [[ $parallelism_limit -eq 1 ]]; then + parallelism_disabled=true + else + # Could evaluate to <1 + parallelism_limit=$((parallelism_limit)) + fi + + if [[ $parallelism_limit -lt 1 ]]; then + # Suppress warning for edge cases when only 1 CPU available or + # when `--parallelism-ci-cpu-cores=1` and `--parallelism_limit` unset + if [[ $CPU -ne 1 ]]; then + + common::colorify "yellow" "Observed Parallelism limit '$parallelism_limit'." \ + "To avoid possible harm, parallelism set to '1'" + fi + + parallelism_limit=1 + parallelism_disabled=true + fi + + local pids=() + + mapfile -t dir_paths_unique < <(echo "${dir_paths[@]}" | tr ' ' '\n' | sort -u) + local length=${#dir_paths_unique[@]} + local last_index=$((${#dir_paths_unique[@]} - 1)) + + local final_exit_code=0 # preserve errexit status shopt -qo errexit && ERREXIT_IS_SET=true # allow hook to continue if exit_code is greater than 0 set +e - local final_exit_code=0 - - # run hook for each path - for dir_path in $(echo "${dir_paths[*]}" | tr ' ' '\n' | sort -u); do - dir_path="${dir_path//__REPLACED__SPACE__/ }" + # run hook for each path in parallel + for ((i = 0; i < length; i++)); do + dir_path="${dir_paths_unique[$i]//__REPLACED__SPACE__/ }" + { + if [[ $change_dir_in_unique_part == false ]]; then + pushd "$dir_path" > /dev/null + fi - if [[ $change_dir_in_unique_part == false ]]; then - pushd "$dir_path" > /dev/null || continue - fi + per_dir_hook_unique_part "$dir_path" "$change_dir_in_unique_part" "$parallelism_disabled" "${args[@]}" + } & + pids+=("$!") - per_dir_hook_unique_part "$dir_path" "$change_dir_in_unique_part" "${args[@]}" + if [[ $parallelism_disabled == true ]] || + [[ $i -ne 0 && $((i % parallelism_limit)) -eq 0 ]] || # don't stop on first iteration when parallelism_limit>1 + [[ $i -eq $last_index ]]; then - local exit_code=$? - if [ $exit_code -ne 0 ]; then - final_exit_code=$exit_code - fi + for pid in "${pids[@]}"; do + # Get the exit code from the background process + local exit_code=0 + wait "$pid" || exit_code=$? - if [[ $change_dir_in_unique_part == false ]]; then - popd > /dev/null + if [ $exit_code -ne 0 ]; then + final_exit_code=$exit_code + fi + done + # Reset pids for next iteration + unset pids fi done @@ -291,14 +416,15 @@ function common::colorify { # Params start # local COLOR="${!1}" - local -r TEXT=$2 + shift + local -r TEXT="$*" # Params end # if [ "$PRE_COMMIT_COLOR" = "never" ]; then COLOR=$RESET fi - echo -e "${COLOR}${TEXT}${RESET}" + echo -e "${COLOR}${TEXT}${RESET}" >&2 } ####################################################################### @@ -307,8 +433,11 @@ function common::colorify { # command_name (string) command that will tun after successful init # dir_path (string) PATH to dir relative to git repo root. # Can be used in error logging +# parallelism_disabled (bool) if true - skip lock mechanism # Globals (init and populate): # TF_INIT_ARGS (array) arguments for `terraform init` command +# TF_PLUGIN_CACHE_DIR (string) user defined env var with name of the directory +# which can't be R/W concurrently # Outputs: # If failed - print out terraform init output ####################################################################### @@ -316,6 +445,7 @@ function common::colorify { function common::terraform_init { local -r command_name=$1 local -r dir_path=$2 + local -r parallelism_disabled=$3 local exit_code=0 local init_output @@ -325,9 +455,32 @@ function common::terraform_init { TF_INIT_ARGS+=("-no-color") fi - if [ ! -d .terraform/modules ] || [ ! -d .terraform/providers ]; then - init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1) - exit_code=$? + recreate_modules=$([[ ! -d .terraform/modules ]] && echo true || echo false) + recreate_providers=$([[ ! -d .terraform/providers ]] && echo true || echo false) + + if [[ $recreate_modules == true || $recreate_providers == true ]]; then + # Plugin cache dir can't be written concurrently or read during write + # https://github.com/hashicorp/terraform/issues/31964 + if [[ -z $TF_PLUGIN_CACHE_DIR || $parallelism_disabled == true ]]; then + init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1) + exit_code=$? + else + # Locking just doesn't work, and the below works quicker instead. Details: + # https://github.com/hashicorp/terraform/issues/31964#issuecomment-1939869453 + for i in {1..10}; do + init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1) + exit_code=$? + + if [ $exit_code -eq 0 ]; then + break + fi + sleep 1 + + common::colorify "green" "Race condition detected. Retrying 'terraform init' command [retry $i]: $dir_path." + [[ $recreate_modules == true ]] && rm -rf .terraform/modules + [[ $recreate_providers == true ]] && rm -rf .terraform/providers + done + fi if [ $exit_code -ne 0 ]; then common::colorify "red" "'terraform init' failed, '$command_name' skipped: $dir_path" diff --git a/hooks/terraform_checkov.sh b/hooks/terraform_checkov.sh index 03ad208c6..a9603afd9 100755 --- a/hooks/terraform_checkov.sh +++ b/hooks/terraform_checkov.sh @@ -34,6 +34,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -43,7 +44,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") checkov -d . "${args[@]}" diff --git a/hooks/terraform_fmt.sh b/hooks/terraform_fmt.sh index bb0b327d1..727635caa 100755 --- a/hooks/terraform_fmt.sh +++ b/hooks/terraform_fmt.sh @@ -31,6 +31,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -40,7 +41,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/terraform_providers_lock.sh b/hooks/terraform_providers_lock.sh index d85a794f8..bf2aa2084 100755 --- a/hooks/terraform_providers_lock.sh +++ b/hooks/terraform_providers_lock.sh @@ -85,6 +85,7 @@ function lockfile_contains_all_needed_sha { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -93,7 +94,8 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") local platforms_count=0 @@ -136,7 +138,7 @@ function per_dir_hook_unique_part { common::colorify "yellow" "DEPRECATION NOTICE: We introduced '--mode' flag for this hook. Check migration instructions at https://github.com/antonbabenko/pre-commit-terraform#terraform_providers_lock " - common::terraform_init 'terraform providers lock' "$dir_path" || { + common::terraform_init 'terraform providers lock' "$dir_path" "$parallelism_disabled" || { exit_code=$? return $exit_code } diff --git a/hooks/terraform_tflint.sh b/hooks/terraform_tflint.sh index f26201a3c..9c8373676 100755 --- a/hooks/terraform_tflint.sh +++ b/hooks/terraform_tflint.sh @@ -44,6 +44,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -51,7 +52,9 @@ function main { function per_dir_hook_unique_part { local -r dir_path="$1" local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") if [ "$change_dir_in_unique_part" == "delegate_chdir" ]; then diff --git a/hooks/terraform_tfsec.sh b/hooks/terraform_tfsec.sh index d05ed4241..75966bf8c 100755 --- a/hooks/terraform_tfsec.sh +++ b/hooks/terraform_tfsec.sh @@ -37,6 +37,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -46,7 +47,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/terraform_trivy.sh b/hooks/terraform_trivy.sh index dc205601a..fd9a3203a 100755 --- a/hooks/terraform_trivy.sh +++ b/hooks/terraform_trivy.sh @@ -29,6 +29,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -38,7 +39,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/terraform_validate.sh b/hooks/terraform_validate.sh index 2f3795a72..9a6c50fce 100755 --- a/hooks/terraform_validate.sh +++ b/hooks/terraform_validate.sh @@ -75,6 +75,7 @@ function match_validate_errors { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -83,7 +84,8 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") local exit_code @@ -121,7 +123,7 @@ function per_dir_hook_unique_part { # In case `terraform validate` failed to execute # - check is simple `terraform init` will help - common::terraform_init 'terraform validate' "$dir_path" || { + common::terraform_init 'terraform validate' "$dir_path" "$parallelism_disabled" || { exit_code=$? return $exit_code } @@ -150,7 +152,7 @@ function per_dir_hook_unique_part { common::colorify "yellow" "Re-validating: $dir_path" - common::terraform_init 'terraform validate' "$dir_path" || { + common::terraform_init 'terraform validate' "$dir_path" "$parallelism_disabled" || { exit_code=$? return $exit_code } diff --git a/hooks/terragrunt_fmt.sh b/hooks/terragrunt_fmt.sh index 7c78b9233..1effbf47b 100755 --- a/hooks/terragrunt_fmt.sh +++ b/hooks/terragrunt_fmt.sh @@ -27,6 +27,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -36,7 +37,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/terragrunt_validate.sh b/hooks/terragrunt_validate.sh index 15e203be7..8fafada0a 100755 --- a/hooks/terragrunt_validate.sh +++ b/hooks/terragrunt_validate.sh @@ -27,6 +27,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -36,7 +37,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/terrascan.sh b/hooks/terrascan.sh index ac040b93e..0c1447bd4 100755 --- a/hooks/terrascan.sh +++ b/hooks/terrascan.sh @@ -27,6 +27,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -36,7 +37,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook diff --git a/hooks/tfupdate.sh b/hooks/tfupdate.sh index d9a482917..9397cfe94 100755 --- a/hooks/tfupdate.sh +++ b/hooks/tfupdate.sh @@ -37,6 +37,7 @@ function main { # change_dir_in_unique_part (string/false) Modifier which creates # possibilities to use non-common chdir strategies. # Availability depends on hook. +# parallelism_disabled (bool) if true - skip lock mechanism # args (array) arguments that configure wrapped tool behavior # Outputs: # If failed - print out hook checks status @@ -46,7 +47,9 @@ function per_dir_hook_unique_part { local -r dir_path="$1" # shellcheck disable=SC2034 # Unused var. local -r change_dir_in_unique_part="$2" - shift 2 + # shellcheck disable=SC2034 # Unused var. + local -r parallelism_disabled="$3" + shift 3 local -a -r args=("$@") # pass the arguments to hook