Fix: run terraform validate on modules again (#20230)

Only running `terraform validate` on deployments is unexpected for
users. See #20222 . Running
`terraform validate` on all modules might lead to errors with Terraform,
but those are predictable and understandable by users.
This MR:
- Runs `terraform validate` on `terraform_modules` again
- Adds the `skip_terraform_validate` field for `terraform_modules` which
cannot be `validate`d by Terraform
- Adds some documentation for the Terraform backend

It leaves open the possibility of validating both
`terraform_deployment`s and `terraform_module`s. Hopefully we can work
out something like the Helm backend's handling of kubeconform.

---------

Co-authored-by: Benjy Weinberger <benjyw@gmail.com>

docs/markdown/Terraform/terraform-overview.md

@@ -0,0 +1,124 @@
+---
+title: "Terraform Overview"
+slug: "terraform-overview"
+hidden: false
+createdAt: "2023-11-22T17:00:00.000Z"
+---
+> 🚧 Terraform support is in alpha stage
+> 
+> Pants is currently building support for developing and deploying Terraform. Simple use cases might be supported, but many options are missing. 
+> 
+> Please share feedback for what you need to use Pants with your Terraform modules and deployments by either [opening a GitHub issue](https://github.com/pantsbuild/pants/issues/new/choose) or [joining our Slack](doc:getting-help)!
+
+Initial setup
+=============
+
+First, activate the relevant backend in `pants.toml`:
+
+```toml pants.toml
+[GLOBAL]
+backend_packages = [
+  ...
+  "pants.backend.experimental.terraform",
+  ...
+]
+```
+
+The Terraform backend also needs Python to run Pants's analysers. The setting `[python].interpreter_constraints` will need to be set.
+
+Adding Terraform targets
+------------------------
+
+The Terraform backend has 2 target types:
+- `terraform_module` for Terraform source code
+- `terraform_deployment` for deployments that can be deployed with the `experimental-deploy` goal
+
+### Modules
+
+The `tailor` goal will automatically generate `terraform_module` targets. Run [`pants tailor ::`](doc:initial-configuration#5-generate-build-files). For example:
+
+```
+❯ pants tailor ::
+Created src/terraform/root/BUILD:
+  - Add terraform_module target root
+```
+
+### Deployments
+
+`terraform_deployments` must be manually created. The deployment points to a `terraform_module` target as its `root_module` field. This module will be the "root" module that Terraform operations will be run on. You can reference vars files with the `var_files` field. You can have multiple deployments reference the same module:
+
+
+```
+terraform_module(name="root")
+terraform_deployment(name="prod", root_module=":root", var_files=["prod.tfvars"]) 
+terraform_deployment(name="test", root_module=":root", var_files=["test.tfvars"]) 
+```
+
+### Lockfiles
+
+Automatic lockfile management is currently in progress. You can include lockfiles manually as a dependency:
+
+```
+terraform_deployment(name="prod", root_module=":root", dependencies=[":lockfile"])
+file(name="lockfile", source=".terraform.lock.hcl")
+```
+
+Basic Operations
+----------------
+
+### Formatting
+
+Run `terraform fmt` as part of the `fix`, `fmt`, or `lint` goals.
+
+```
+pants fix ::
+[INFO] Completed: pants.backend.terraform.lint.tffmt.tffmt.tffmt_fmt - terraform-fmt made no changes.
+
+✓ terraform-fmt made no changes.
+```
+
+### Validate
+
+Run `terraform validate` as part of the `check` goal.
+
+```
+pants check ::
+[INFO] Completed: pants.backend.terraform.goals.check.terraform_check - terraform-validate succeeded.
+Success! The configuration is valid.
+
+✓ terraform-validate succeeded.
+
+```
+
+`terraform validate` isn't valid for all Terraform modules. Some child modules, in particular those using aliased providers, need to have their providers provided by a "root" module. You can opt these modules out of `validate` by setting `skip_terraform_validate=True`. For example:
+
+```
+terraform_module(skip_terraform_validate=True)
+```
+
+### Deploying
+
+> 🚧 Terraform deployment support is in alpha stage
+> 
+> Many options and features aren't supported yet. 
+> Local state backends aren't supported.
+
+
+Run `terraform apply` as part of the `experimental-deploy` goal. The process is run interactively, so you will be prompted for variables and confirmation as usual.
+
+```
+pants experimental-deploy ::
+[INFO] Deploying targets...
+--- 8< ---
+Do you want to perform these actions?
+  Terraform will perform the actions described above.
+  Only 'yes' will be accepted to approve.
+
+  Enter a value: yes
+--- 8< ---
+Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
+
+✓ testprojects/src/terraform/root:root deployed
+```
+
+You can set auto approve by adding `-auto-approve` to the `[download-terraform].args` setting in `pants.toml`. You can also set it for a single pants invocation with `--download-terraform-args='-auto-approve'`, for example `pants experimental-deploy "--download-terraform-args='-auto-approve'"`.

docs/markdown/Writing Plugins/common-plugin-tasks/plugins-general-subsystems.md

@@ -0,0 +1,50 @@
+---
+title: "Common subsystem tasks"
+slug: "plugins-common-subsystem"
+excerpt: "Common tasks for Subsystems"
+hidden: false
+createdAt: "2023-11-22T17:00:00.000Z"
+---
+Skipping individual targets
+---------------------------
+
+Many subsystems allow skipping specific targets. For example, you might have Python files that you want to not typecheck with mypy. In Pants, this is achieved with a `skip_*` field on the target. This is simple to implement.
+
+1. Create a field for skipping your tool
+
+```python
+from pants.engine.target import BoolField
+
+class SkipFortranLintField(BoolField):
+	alias = "skip_fortran_lint"
+	default = False
+	help = "If true, don't run fortran-lint on this target's code."
+```
+
+2. Register this field on the appropriate targets.
+
+```python
+def rules():
+	return [
+		FortranSourceTarget.register_plugin_field(SkipFortranLintField),
+	]
+```
+
+3. Add this field as part of your subsystems `opt_out` method:
+
+```python
+from dataclasses import dataclass
+
+from pants.engine.target import FieldSet, Target
+
+
+@dataclass
+class FortranLintFieldSet(FieldSet):
+    required_fields = (FortranSourceField,)
+
+    source: FortranSourceField
+
+    @classmethod
+    def opt_out(cls, tgt: Target) -> bool:
+        return tgt.get(SkipFortranLintField).value
+```

src/python/pants/backend/terraform/goals/check.py

@@ -1,12 +1,23 @@
 # Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
+from dataclasses import dataclass
+from typing import Union
+
 from pants.backend.terraform.dependencies import TerraformInitRequest, TerraformInitResponse
-from pants.backend.terraform.target_types import TerraformDeploymentFieldSet
+from pants.backend.terraform.target_types import (
+    TerraformBackendConfigField,
+    TerraformDeploymentFieldSet,
+    TerraformDeploymentTarget,
+    TerraformFieldSet,
+    TerraformModuleTarget,
+    TerraformRootModuleField,
+)
 from pants.backend.terraform.tool import TerraformProcess
 from pants.core.goals.check import CheckRequest, CheckResult, CheckResults
 from pants.engine.internals.selectors import Get, MultiGet
 from pants.engine.process import FallibleProcessResult
 from pants.engine.rules import collect_rules, rule
+from pants.engine.target import BoolField, Target
 from pants.engine.unions import UnionRule
 from pants.option.option_types import SkipOption
 from pants.option.subsystem import Subsystem
@@ -21,11 +32,41 @@ class TerraformValidateSubsystem(Subsystem):
     skip = SkipOption("check")
 
 
+class SkipTerraformValidateField(BoolField):
+    alias = "skip_terraform_validate"
+    default = False
+    help = "If true, don't run `terraform validate` on this target's code. If this target is a module, `terraform validate might still be run on a `terraform_deployment that references this module."
+
+
+@dataclass(frozen=True)
+class TerraformValidateFieldSet(TerraformFieldSet):
+    @classmethod
+    def opt_out(cls, tgt: Target) -> bool:
+        return tgt.get(SkipTerraformValidateField).value
+
+
 class TerraformCheckRequest(CheckRequest):
-    field_set_type = TerraformDeploymentFieldSet
+    field_set_type = TerraformValidateFieldSet
     tool_name = TerraformValidateSubsystem.options_scope
 
 
+def terraform_fieldset_to_init_request(
+    terraform_fieldset: Union[TerraformDeploymentFieldSet, TerraformFieldSet]
+) -> TerraformInitRequest:
+    if isinstance(terraform_fieldset, TerraformDeploymentFieldSet):
+        deployment = terraform_fieldset
+        return TerraformInitRequest(
+            deployment.root_module, deployment.backend_config, deployment.dependencies
+        )
+    if isinstance(terraform_fieldset, TerraformFieldSet):
+        module = terraform_fieldset
+        return TerraformInitRequest(
+            TerraformRootModuleField(module.address.spec, module.address),
+            TerraformBackendConfigField(None, module.address),
+            module.dependencies,
+        )
+
+
 @rule
 async def terraform_check(
     request: TerraformCheckRequest, subsystem: TerraformValidateSubsystem
@@ -36,9 +77,8 @@ async def terraform_check(
     initialised_terraforms = await MultiGet(
         Get(
             TerraformInitResponse,
-            TerraformInitRequest(
-                deployment.root_module, deployment.backend_config, deployment.dependencies
-            ),
+            TerraformInitRequest,
+            terraform_fieldset_to_init_request(deployment),
         )
         for deployment in request.field_sets
     )
@@ -71,5 +111,7 @@ async def terraform_check(
 def rules():
     return (
         *collect_rules(),
+        TerraformDeploymentTarget.register_plugin_field(SkipTerraformValidateField),
+        TerraformModuleTarget.register_plugin_field(SkipTerraformValidateField),
         UnionRule(CheckRequest, TerraformCheckRequest),
     )

src/python/pants/backend/terraform/goals/check_test.py

@@ -11,8 +11,8 @@
 from pants.backend.terraform.goals import check
 from pants.backend.terraform.goals.check import TerraformCheckRequest
 from pants.backend.terraform.target_types import (
-    TerraformDeploymentFieldSet,
     TerraformDeploymentTarget,
+    TerraformFieldSet,
     TerraformModuleTarget,
 )
 from pants.core.goals.check import CheckResult, CheckResults
@@ -85,7 +85,7 @@ def make_target(
     }
     files.update({source_file.path: source_file.content.decode() for source_file in source_files})
     rule_runner.write_files(files)
-    return rule_runner.get_target(Address("", target_name=target_name))
+    return rule_runner.get_target(Address("", target_name=f"{target_name}mod"))
 
 
 def run_terraform_validate(
@@ -95,7 +95,7 @@ def run_terraform_validate(
     args: list[str] | None = None,
 ) -> Sequence[CheckResult]:
     rule_runner.set_options(args or ())
-    field_sets = [TerraformDeploymentFieldSet.create(tgt) for tgt in targets]
+    field_sets = [TerraformFieldSet.create(tgt) for tgt in targets]
     check_results = rule_runner.request(CheckResults, [TerraformCheckRequest(field_sets)])
     return check_results.results
 
@@ -155,8 +155,8 @@ def test_multiple_targets(rule_runner: RuleRunner) -> None:
     )
 
     targets = [
-        rule_runner.get_target(Address("", target_name="tgt_good")),
-        rule_runner.get_target(Address("", target_name="tgt_bad")),
+        rule_runner.get_target(Address("", target_name="good")),
+        rule_runner.get_target(Address("", target_name="bad")),
     ]
 
     check_results = run_terraform_validate(rule_runner, targets)
@@ -206,7 +206,7 @@ def test_in_folder(rule_runner: RuleRunner) -> None:
         ),
     }
     rule_runner.write_files(files)
-    target = rule_runner.get_target(Address("folder", target_name=target_name))
+    target = rule_runner.get_target(Address("folder", target_name="mod0"))
 
     check_results = run_terraform_validate(rule_runner, [target])
     assert check_results[0].exit_code == 0
@@ -251,7 +251,7 @@ def make_terraform_module(version: str) -> Dict[str, str]:
 
     rule_runner.write_files(files)
     targets = [
-        rule_runner.get_target(Address(folder, target_name=target_name))
+        rule_runner.get_target(Address(folder, target_name="mod"))
         for folder in (f"folder{version}" for version in versions)
     ]