Skip to content

content(what-is): expand the cloud infrastructure testing explainer#19146

Merged
CamSoper merged 4 commits into
masterfrom
aleventer/what-is-cloud-infra-testing-rewrite
May 19, 2026
Merged

content(what-is): expand the cloud infrastructure testing explainer#19146
CamSoper merged 4 commits into
masterfrom
aleventer/what-is-cloud-infra-testing-rewrite

Conversation

@alexleventer
Copy link
Copy Markdown
Contributor

@alexleventer alexleventer commented May 18, 2026

Summary

Rewrites content/what-is/how-to-step-up-cloud-infrastructure-testing.md from a 4-section overview into a deeper testing-program reference. Body grows from ~70 lines to ~180 well-structured lines.

What changed

  • Opening definition — quotable one-paragraph definition of IaC testing followed by a short lead-in about the testing pyramid mindset.
  • Why test IaC — three concrete drivers (misconfiguration dominance, scale of manual review, rollback cost).
  • Layers of testing comparison table — static analysis, unit, property, integration, security; each row noting whether real resources are provisioned and runtime.
  • Per-layer sections for unit, property, integration, and security/policy testing, each with concrete example assertions and a link to the relevant Pulumi guide.
  • CI/CD pipeline placement — what runs on commit vs. PR vs. merge vs. promotion.
  • Tools table — unit runners, static scanners (Checkov, tfsec, etc.), policy-as-code, integration tools, cloud emulators (LocalStack, Moto), image/dep scanners, chaos engineering.
  • Pulumi-specific section — automation API, CrossGuard, unit-test mocks, GitHub Actions integration.
  • FAQ — ten doubt-removers including unit vs. integration, property testing, cost control, TDD for IaC, compliance evidence.
  • Learn more — cross-links to IaC, DevOps, cloud security, configuration management.

Test plan

  • make serve; visit /what-is/how-to-step-up-cloud-infrastructure-testing/ and confirm tables and headings render correctly
  • Spot-check cross-links to /docs/iac/using-pulumi/testing/, /docs/insights/policy/, /docs/iac/packages-and-automation/automation-api/, and the other what-is pages
  • CI lint + pinned review

🤖 Generated with Claude Code

@alexleventer @claude
content(what-is): expand the cloud infrastructure testing explainer
09410e4 
Rewrites content/what-is/how-to-step-up-cloud-infrastructure-testing.md
into a deeper reference for testing infrastructure as code.

New structure:
- Bold quotable definition + question-driven TOC.
- Why test IaC (misconfig is the dominant breach source, manual
  review doesn't scale, rollback is expensive).
- Layers-of-testing comparison table: static analysis, unit,
  property, integration, security.
- Dedicated sections for unit, property, integration, and
  security/policy testing — each with concrete assertions to make
  and link to Pulumi guides.
- Where each layer fits in a CI/CD pipeline.
- Tool table covering unit, static scan, policy as code, property/
  integration, cloud emulation, image scanning, chaos engineering.
- Pulumi-support section calling out automation API, CrossGuard,
  test mocks, GitHub Actions integration.
- Ten FAQ entries: unit vs integration, property testing, cost
  control, when security tests run, TDD for IaC, policy as code,
  compliance evidence, brownfield adoption.
- Learn-more cross-links to IaC, IaC for DevOps, DevOps, cloud
  security, configuration management.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@pulumi-bot
Copy link
Copy Markdown
Collaborator

pulumi-bot commented May 18, 2026 •
edited
Loading

Pre-merge Review — Last updated 2026-05-19T18:15:00Z

Tip

Summary: This PR substantially expands /what-is/how-to-step-up-cloud-infrastructure-testing/ — a what-is explainer that parallels existing pages like what-is-infrastructure-as-code.md and what-is-cloud-security.md. All 7 blockers identified in the first review have been addressed: four broken internal links corrected to canonical paths, tfsec replaced with Trivy throughout (tfsec was archived into Trivy in 2023), CrossGuard Java mis-attribution removed, and the SOC 2/HIPAA/PCI framing softened from "increasingly prefer" to "routinely accept." Per @CamSoper's request, all deprecated "CrossGuard" brand references have been replaced with "Pulumi Policies" throughout the page. The page is clean and ready for human final approval.

Review confidence:

Dimension Level Notes
mechanics MEDIUM Hugo build was skipped (content-only PR per .hugo-build.json); internal links were manually grepped against canonical paths and alias declarations
facts HIGH All 41 extracted claims have verdicts; broken-link findings independently confirmed against the live content tree; CrossGuard→Pulumi Policies rename verified as factually consistent
Investigation log
  • Cross-sibling reads: not run (not in a templated section)
  • External claim verification: 22 of 41 claims verified (6 unverifiable, 6 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 29 Pass 1, 0 Pass 2, 12 Pass 3 (verified 6, contradicted 2, unverifiable 4).
  • Cited-claim spot-checks: not run (no cited claims)
  • Frontmatter sweep: ran on body + meta_desc
  • Temporal-trigger sweep: ran (recency words present in diff; spot-check in-review)
  • Code execution: not run (no static/programs/ change)
  • Code-examples checks: not run (no fenced code blocks in content files)
  • Editorial-balance pass: not run (not under content/blog/)
🚨 Outstanding ⚠️ Low-confidence 💡 Pre-existing ✅ Resolved
0 0 0 8

🔍 Verification trail

42 claims extracted · 22 verified · 6 unverifiable · 7 contradicted
  • L32 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "When infrastructure lives in source control as infrastructure as code (IaC), the same engineering disciplines that catch bugs in applications can catch bugs in…" → ✅ verified (evidence: The exact sentence appears verbatim in the file: "When infrastructure lives in source control as infrastructure as code (IaC), the same engineering disciplines that catch bugs in applications can catch bugs in VPCs, IAM policies, Kubernete…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L32 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "**Cloud infrastructure testing is the practice of validating the code that defines your cloud resources the same way you'd validate application code: with unit…" → ➖ not-a-claim (evidence: The bolded text at L32 is the PR author's own definition of "cloud infrastructure testing" written in their own document. It is a faithful description of the author's own design/framing, not a falsifiable factual assertion attributed to a third-party source.)
  • L34 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Most teams that adopt IaC stop at deploy-and-verify: spin up the resources, click around, hope for the best. That's the infrastructure equivalent of 'no automa…" → ➖ not-a-claim (evidence: The text is the PR author's own editorial positioning — an opinion about common IaC team behavior and an analogy to the testing pyramid. It is not a falsifiable factual assertion attributed to a third-party source.)
  • L53 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Misconfigurations are the dominant source of cloud incidents, with public buckets, overly broad IAM, open security groups, and missing encryption accounting fo…" → ✅ verified (framing: narrowed — sources confirm misconfigurations are a top/leading cause of cloud breaches and that public buckets, IAM, security groups, and encryption are the main vectors.)
  • L54 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* Manual review doesn't scale. A typical cloud-native app spans hundreds of resources changing daily. No human reviewer reliably catches a regression like…" → ➖ not-a-claim (evidence: This is an editorial/opinion statement made by the PR author about why manual review doesn't scale for cloud-native apps. No falsifiable factual assertion tied to a third-party source.)
  • L55 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* Rollback is expensive. Reverting an infrastructure change can cascade: dropped database, deleted secrets, broken DNS. Catching the issue in CI is dramati…" → ➖ not-a-claim (evidence: The statement is the PR author's own editorial framing of a general DevOps principle. Not attributed to any third-party source.)
  • L63 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Property tests do not provision real resources (or use transient resources) and have a typical runtime of seconds to minutes." (also L67) → ✅ verified (evidence: The file's comparison table explicitly states for Property tests: "Provisions real resources?" → "No (or transient)" and "Typical runtime" → "Seconds to minutes".)
  • L73 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi programs can be unit-tested using Jest/Vitest for TypeScript, pytest for Python, go test for Go, xUnit for C#, and JUnit for Java." → ✅ verified (evidence: The file states: "Pulumi programs can be unit-tested using the standard test runners for the language you wrote them in (Jest/Vitest for TypeScript, pytest for Python, go test for Go, xUnit for C#, JUnit for Java).")
  • L82 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "IaC unit tests run in seconds." → ✅ verified (evidence: The file states: "Unit tests run in seconds and produce the tightest feedback loop.")
  • L84 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi has a unit testing guide at /docs/iac/using-pulumi/testing/unit/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL is not declared as an alias on content/docs/iac/guides/testing/unit.md; canonical URL is /docs/iac/guides/testing/unit/]
  • L88 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Property tests run after a pulumi preview (or equivalent) produces a plan, against the planned resource graph or the freshly-deployed real resources." → ❌ contradicted (framing: shifted — claim frames property tests as running after a preview against a "planned resource graph," but the source says they run during actual infrastructure provisioning; source: repo:content/docs/iac/guides/testing/_index.md)
  • L88 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Unlike integration tests, property tests assert on specific resource properties rather than end-to-end behavior." → ✅ verified (framing: strengthened — claim narrows the broader source contrast ("resource-level assertions" vs "end-to-end behavior") to a specific framing about "specific resource properties".)
  • L95 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* 'Every service has logging enabled and shipped to the central log account.'" → ➖ not-a-claim (evidence: A bullet point in the PR author's own documentation describing a desired infrastructure property. Not a falsifiable assertion about a third-party system.)
  • L97 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "In Pulumi, property-level checks are usually written as policy as code using Pulumi Policies, and the same policy code runs against pulumi preview (blocking the merge) and as a deploy-time gate." → ✅ verified (evidence: File states "In Pulumi, these checks are usually written as policy as code using Pulumi Policies. The same policy code runs against pulumi preview (blocking the merge) and as a deploy-time gate."; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L101 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Integration tests deploy real cloud resources into an ephemeral environment, run end-to-end checks, then tear the environment down." → ✅ verified (evidence: Verbatim match in the file.)
  • L111 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Integration tests are the slowest and most expensive layer of infrastructure testing because they consume cloud resources." → ✅ verified (framing: strengthened — sources confirm integration tests are slower/more expensive than other layers because they provision real cloud resources.)
  • L113 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi has an integration testing guide at /docs/iac/using-pulumi/testing/integration/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL is not declared as an alias on content/docs/iac/guides/testing/integration/_index.md; canonical URL is /docs/iac/guides/testing/integration/]
  • L119 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Tools like Checkov, tfsec, Terrascan, and Snyk IaC scan Terraform, CloudFormation, and Pulumi outputs against a built-in ruleset." → ❌ contradicted (framing: narrowed — claim broadens Snyk IaC's supported frameworks to include Pulumi; source supports only Terraform, CloudFormation, Kubernetes, and ARM. Also, tfsec is archived; source: https://docs.snyk.io/scan-with-snyk/snyk-iac)
  • L121 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi Policies run during pulumi preview and pulumi up, so a non-compliant change can't get past CI." → ✅ verified (evidence: The /docs/insights/policy/ page states: "Preventative: Validates Pulumi stack resources during pulumi preview and pulumi up, blocking deployments when violations are detected.")
  • L121 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi Policies lets you author policies in TypeScript/JavaScript, Python, or OPA's Rego against the actual Pulumi resource model." → ✅ verified (evidence: Official /docs/insights/policy/ page states: "Policies can be written in TypeScript/JavaScript (Node.js), Python, or OPA (Rego)" — Java correctly excluded; source: repo:content/docs/insights/policy/_index.md)
  • L129 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Whatever the mix, the consistent rule is shift left: every security check that can run in CI should, and the deploy gate should fail closed." → ➖ not-a-claim (evidence: General best-practice editorial assertion authored by the PR author — not attributed to any third-party source.)
  • L146 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Unit testing for IaC uses Jest, Vitest, pytest, go test, xUnit, and JUnit as standard test runners for the language IaC is written in." → ✅ verified (evidence: File confirms all six test runners at the unit testing section.)
  • L147 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Checkov, Terrascan, Trivy, and Snyk IaC are representative tools for static IaC scanning." → ✅ verified (evidence: File explicitly names all four tools in the static scanning section.)
  • L148 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi Policies, Open Policy Agent (OPA), and HashiCorp Sentinel are representative tools for policy as code." → ✅ verified (framing: strengthened — the source confirms Pulumi Policies and OPA (Rego); Sentinel is a real HashiCorp product and legitimate representative tool; source: repo:content/docs/insights/policy/_index.md)
  • L149 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi automation API, Terratest, and Kitchen-Terraform are representative tools for property and integration testing." → ✅ verified (framing: strengthened — the claim groups all three tools under "property and integration testing," but sources show Pulumi Automation API is specifically for integration testing.)
  • L150 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "LocalStack emulates AWS, Moto emulates AWS, and Azurite emulates Azure for cloud emulation in testing." → ✅ verified (framing: strengthened — Azurite specifically emulates Azure Storage services rather than all of Azure.)
  • L151 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Trivy, Snyk, Anchore, and Grype are representative tools for image and dependency scanning." → ✅ verified (framing: strengthened — Grype is Anchore's open-source scanner; source: confirmed by multiple sources.)
  • L152 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Gremlin, AWS Fault Injection Simulator, and Chaos Mesh are representative tools for chaos engineering." → ✅ verified (evidence: Multiple authoritative sources confirm all three tools; source: https://www.gremlin.com/community/tutorials/chaos-engineering-tools-comparison)
  • L160 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi supports writing programs in TypeScript, Python, Go, C#, Java, or YAML." → ❌ contradicted (framing: shifted — claim lists "TypeScript, Python, Go, C#, Java, or YAML" but the authoritative source lists "TypeScript, JavaScript, Python, Go, .NET, Java, and YAML"; source: repo:content/docs/iac/languages-sdks/_index.md)
  • L161 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi's test mocks documentation is at /docs/iac/using-pulumi/testing/unit/." → ❌ contradicted (in-review link check: /docs/iac/using-pulumi/testing/unit/ is not declared as an alias on the unit-testing page; canonical URL is /docs/iac/guides/testing/unit/; source: repo:content/docs/iac/guides/testing/unit.md)
  • L161-164 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner, so integration tests can deploy and tear down ephemeral stacks p…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L162 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner, so integration tests can deploy and tear down ephemeral stacks p…" → ✅ verified (framing: strengthened — claim narrows the general "programmatic interface for driving Pulumi programs" to the specific test-runner/ephemeral-stack use case; source: gh api repos/pulumi/pulumi/contents/sdk/go/auto/README.md)
  • L163 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi Policies run during preview and update, blocking changes that violate organizational rules." → ✅ verified (evidence: The /docs/insights/policy/ page states: "Preventative: Validates Pulumi stack resources during pulumi preview and pulumi up, blocking deployments when violations are detected.")
  • L164 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi runs in every major CI/CD platform via the GitHub Actions integration or any other system that can run a CLI." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL /docs/iac/packages-and-automation/github-actions/ is not declared as an alias on content/docs/iac/guides/continuous-delivery/github-actions.md; canonical URL is /docs/iac/guides/continuous-delivery/github-actions/]
  • L166 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi supports provisioning and testing infrastructure as code in TypeScript, Python, Go, C#, Java, or YAML." → ❌ contradicted (framing: shifted — claim lists "TypeScript, Python, Go, C#, Java, or YAML" but the source lists "TypeScript, JavaScript, Python, Go, .NET, Java, and YAML"; source: repo:content/docs/iac/languages-sdks/_index.md)
  • L176 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "IaC unit tests run in memory with mocked cloud responses and are fast (seconds), but only test the logic of IaC code." → ✅ verified (evidence: The document states: "Unit tests verify the logic of your IaC programs without calling a cloud provider. External dependencies are replaced with mocks…")
  • L184 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "In Pulumi, property tests are typically written as Pulumi policies, and policy documentation is at /docs/insights/policy/." → ✅ verified (evidence: File states "In Pulumi, property tests are typically written as Pulumi policies"; policy docs confirmed at that path; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md and repo:content/docs/insights/policy/_index.md)
  • L204 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Compliance frameworks such as SOC 2, HIPAA, and PCI accept IaC test results as evidence, and increasingly they prefer it." → ❌ contradicted (framing: narrowed — claim broadens the source's "IaC test results can serve as compliance evidence" to "frameworks increasingly prefer them"; no authoritative source supports that framing; source: WebSearch)
  • L204 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "A Pulumi Policies run produces a concrete artifact that proves a control was enforced on a specific change at a specific time." → 🤷 unverifiable (evidence: Official Pulumi Policies documentation describes policy enforcement and violation reporting, but no authoritative source describes a run as producing a "concrete artifact" proving enforcement at a specific time in auditor-preferred terms.)
  • L208 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Start with the cheapest layer that produces the most value: static scanning and policy as code in advisory mode." → ➖ not-a-claim (evidence: Prescriptive methodology recommendation authored by the PR author — contains no falsifiable assertion about a third-party system.)
  • L212 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi automation API documentation is at /docs/iac/packages-and-automation/automation-api/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L216-220 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* What is Infrastructure as Code (IaC)?" → ✅ verified (evidence: The file content/what-is/what-is-infrastructure-as-code.md exists with title: "What is Infrastructure as Code (IaC)?", matching the link text and path.)

📋 Triaged verifier findings

I double-checked these and realized they weren't real findings — click to expand

  • [L88] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Property tests run after a pulumi preview (or equivalent) produces a plan, against the planned resource graph or the freshly-deployed real resources." — Spurious: The verifier compared the claim against a partial source passage that emphasized only the during-deployment mode. The PR's framing actually covers both modes explicitly ("the planned resource graph or the freshly-deployed real resources"), which matches Pulumi Policies' documented behavior of running at both pulumi preview (against the plan) and pulumi up (against deployed resources).

  • [L160] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Write Pulumi programs in TypeScript, Python, Go, C#, Java, or YAML" — Spurious: The verifier strictly compared this to the canonical languages-SDK summary ("TypeScript, JavaScript, Python, Go, .NET, Java, and YAML") and flagged JavaScript-missing and "C#" vs ".NET". This 6-language shorthand (TS/Python/Go/C#/Java/YAML) is used widely throughout Pulumi's own marketing and docs prose — it's a stylistic convention, not a factual error.

  • [L166] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "to provision and test infrastructure as code in TypeScript, Python, Go, C#, Java, or YAML" — Spurious: Same shorthand as L160 in the closing CTA. Same reasoning applies.

  • [L161-164] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner…" — Spurious: Verifier ran out of turns on a duplicate; the L162 entry for the same claim verified successfully against the Automation API README (which lists integration testing as a use case).

  • [L212] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Pulumi automation API documentation is at /docs/iac/packages-and-automation/automation-api/." — Spurious: The URL resolves via alias. content/docs/iac/automation-api/_index.md declares /docs/iac/packages-and-automation/automation-api/ in its aliases: block, so this link works (canonical URL is /docs/iac/automation-api/ but the alias is intentional).

🚨 Outstanding in this PR

No outstanding findings.

⚠️ Low-confidence

No low-confidence findings.

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

  • [L84] Broken link to the unit testing guide — /docs/iac/using-pulumi/testing/unit/ corrected to /docs/iac/guides/testing/unit/ (resolved in 7d8d32c)

  • [L113] Broken link to the integration testing guide — /docs/iac/using-pulumi/testing/integration/ corrected to /docs/iac/guides/testing/integration/ (resolved in 7d8d32c)

  • [L119] Static-scanner sentence listed archived tfsec and claimed Snyk IaC scans Pulumi outputs (it doesn't) — replaced tfsec with Trivy throughout, narrowed format list to Terraform/CloudFormation/Kubernetes, added note that Checkov has a Pulumi-output mode (resolved in 7d8d32c)

  • [L121] CrossGuard policy authoring listed Java as a supported language (it isn't) — corrected to TypeScript/JavaScript, Python, OPA Rego; added parenthetical clarifying Pulumi Policies apply to stacks in any supported language including Go, .NET, and Java (resolved in 7d8d32c)

  • [L161] Broken link to test-mocks docs — /docs/iac/using-pulumi/testing/unit/ corrected to /docs/iac/guides/testing/unit/ (resolved in 7d8d32c)

  • [L164] Broken link to GitHub Actions integration — /docs/iac/packages-and-automation/github-actions/ corrected to /docs/iac/guides/continuous-delivery/github-actions/ (resolved in 7d8d32c)

  • [L204] "Yes, and increasingly they prefer it" / "Auditors much prefer that" framing overstated what compliance frameworks require — softened to "SOC 2, HIPAA, and PCI DSS audits routinely accept IaC test output and policy-as-code run logs as evidence that a control is enforced" (resolved in 7d8d32c)

  • [L204 ⚠️] Unverifiable "auditors much prefer" claim addressed by the same L204 rewrite (resolved in 7d8d32c)

📜 Review history

  • 2026-05-19T03:34:25Z — First review: 7 blockers (4 broken internal links to Pulumi guides, CrossGuard-Java mis-attribution, tfsec/Snyk-IaC tooling overreach, "increasingly prefer" framing overreach); 1 low-confidence note (09410e4)
  • 2026-05-19T05:12:00Z — re-reviewed after fix push by @alexleventer; all 7 blockers resolved, 1 low-confidence resolved (1 new commit, 7d8d32c)
  • 2026-05-19T17:49:08Z — author field updated to cam-soper per @CamSoper request; no content changes; 0 outstanding unchanged (1 new commit, eb3539f)
  • 2026-05-19T18:15:00Z — deprecated "CrossGuard" brand replaced with "Pulumi Policies" throughout per @CamSoper request; all factual claims verified consistent under new terminology; 0 outstanding unchanged (1 new commit, 8bc9017)

Need a re-review? Want to dispute a finding? Mention @claude and include #update-review.
(For ad-hoc questions or fixes, just @claude — no hashtag.)

@alexleventer alexleventer marked this pull request as ready for review May 19, 2026 03:31
@github-actions github-actions Bot added review:triaging Claude Triage is currently classifying the PR domain:docs PR touches technical docs review:in-progress Claude review is currently running and removed review:triaging Claude Triage is currently classifying the PR labels May 19, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026 •
edited by pulumi-bot
Loading

Pre-merge Review — Last updated 2026-05-19T17:49:08Z

Tip

Summary: This PR substantially expands /what-is/how-to-step-up-cloud-infrastructure-testing/ — a what-is explainer that parallels existing pages like what-is-infrastructure-as-code.md and what-is-cloud-security.md. All 7 blockers identified in the first review have been addressed: four broken internal links corrected to canonical paths, tfsec replaced with Trivy throughout (tfsec was archived into Trivy in 2023), CrossGuard Java mis-attribution removed, and the SOC 2/HIPAA/PCI framing softened from "increasingly prefer" to "routinely accept." The page is clean and ready for human final approval.

Review confidence:

Dimension Level Notes
mechanics MEDIUM Hugo build was skipped (content-only PR per .hugo-build.json); internal links were manually grepped against canonical paths and alias declarations
facts HIGH All 41 extracted claims have verdicts; broken-link findings independently confirmed against the live content tree
Investigation log
  • Cross-sibling reads: not run (not in a templated section)
  • External claim verification: 22 of 41 claims verified (6 unverifiable, 6 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 29 Pass 1, 0 Pass 2, 12 Pass 3 (verified 6, contradicted 2, unverifiable 4).
  • Cited-claim spot-checks: not run (no cited claims)
  • Frontmatter sweep: ran on body + meta_desc
  • Temporal-trigger sweep: ran (recency words present in diff; spot-check in-review)
  • Code execution: not run (no static/programs/ change)
  • Code-examples checks: not run (no fenced code blocks in content files)
  • Editorial-balance pass: not run (not under content/blog/)
🚨 Outstanding ⚠️ Low-confidence 💡 Pre-existing ✅ Resolved
0 0 0 8

🔍 Verification trail

42 claims extracted · 22 verified · 6 unverifiable · 7 contradicted
  • L32 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "When infrastructure lives in source control as infrastructure as code (IaC), the same engineering disciplines that catch bugs in applications can catch bugs in…" → ✅ verified (evidence: The exact sentence appears verbatim in the file: "When infrastructure lives in source control as infrastructure as code (IaC), the same engineering disciplines that catch bugs in applications can catch bugs in VPCs, IAM policies, Kubernete…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L32 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "**Cloud infrastructure testing is the practice of validating the code that defines your cloud resources the same way you'd validate application code: with unit…" → ➖ not-a-claim (evidence: The bolded text at L32 is the PR author's own definition of "cloud infrastructure testing" written in their own document (how-to-step-up-cloud-infrastructure-testing.md). It is a faithful description of the author's own design/framing, n…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L34 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Most teams that adopt IaC stop at deploy-and-verify: spin up the resources, click around, hope for the best. That's the infrastructure equivalent of 'no automa…" → ➖ not-a-claim (evidence: The text is the PR author's own editorial positioning — an opinion about common IaC team behavior and an analogy to the testing pyramid. It is not a falsifiable factual assertion attributed to a third-party source; it is the author's own d…; source: WebSearch ran query "infrastructure testing pyramid IaC automated testing levels"; confirmed testing pyramid is a well-known concept but the claim itself is the author's own editorial framing, not a third-party-attributed assertion.)
  • L53 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Misconfigurations are the dominant source of cloud incidents, with public buckets, overly broad IAM, open security groups, and missing encryption accounting fo…" → ✅ verified (framing: narrowed — sources confirm misconfigurations are a top/leading cause of cloud breaches and that public buckets, IAM, security groups, and encryption are the ma…; evidence: Multiple authoritative sources confirm misconfigurations are a leading/dominant source of cloud breaches, with public buckets, overly broad IAM, open security groups, and missing encryption as the primary vectors. However, the specific fra…; source: WebSearch ran query "misconfigurations dominant source cloud breaches public buckets IAM security groups encryption statistics"; intuition: The "majority of reported cloud breaches" phrasing is stronger than what the data supports — sources cite misconfigurat…)
  • L54 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* Manual review doesn't scale. A typical cloud-native app spans hundreds of resources changing daily. No human reviewer reliably catches a regression like…" → ➖ not-a-claim (evidence: This is an editorial/opinion statement made by the PR author about why manual review doesn't scale for cloud-native apps. It contains no falsifiable factual assertion tied to a third-party source — it's a general argument/motivation paragr…; source: content/what-is/how-to-step-up-cloud-infrastructure-testing.md L54)
  • L55 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* Rollback is expensive. Reverting an infrastructure change can cascade: dropped database, deleted secrets, broken DNS. Catching the issue in CI is dramati…" → ➖ not-a-claim (evidence: The statement is the PR author's own editorial framing of a general DevOps principle (infrastructure rollback can cascade; catching issues in CI is cheaper than a production incident). It is not attributed to any third-party source and is…; source: WebSearch ran query "infrastructure rollback cascade database secrets DNS incident cost CI testing"; claim is author's own explanatory prose, not a cited third-party assertion.)
  • L63 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Property tests do not provision real resources (or use transient resources) and have a typical runtime of seconds to minutes." (also L67) → ✅ verified (evidence: The file's comparison table explicitly states for Property tests: "Provisions real resources?" → "No (or transient)" and "Typical runtime" → "Seconds to minutes", matching the claim exactly.; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L73 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi programs can be unit-tested using Jest/Vitest for TypeScript, pytest for Python, go test for Go, xUnit for C#, and JUnit for Java." → ✅ verified (evidence: The file at L73 states: "Pulumi programs can be unit-tested using the standard test runners for the language you wrote them in (Jest/Vitest for TypeScript, pytest for Python, go test for Go, xUnit for C#, JUnit for Java)." The claim is a…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L82 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "IaC unit tests run in seconds." → ✅ verified (evidence: The file at the relevant section states: "Unit tests run in seconds and produce the tightest feedback loop." This directly confirms the claim that IaC unit tests run in seconds.; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L84 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi has a unit testing guide at /docs/iac/using-pulumi/testing/unit/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL is not declared as an alias on content/docs/iac/guides/testing/unit.md; canonical URL is /docs/iac/guides/testing/unit/]
  • L88 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Property tests run after a pulumi preview (or equivalent) produces a plan, against the planned resource graph or the freshly-deployed real resources." → ❌ contradicted (framing: shifted — claim frames property tests as running after a preview against a "planned resource graph," but the source says they run during actual infrastructure…; evidence: The Pulumi docs state: "Property Tests run resource-level assertions while infrastructure is being deployed" and "Property tests run inside the Pulumi CLI before and after infrastructure provisioning" — they provision real infrastructure…; source: repo:content/docs/iac/guides/testing/_index.md)
  • L88 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Unlike integration tests, property tests assert on specific resource properties rather than end-to-end behavior." → ✅ verified (framing: strengthened — claim narrows the broader source contrast ("resource-level assertions" vs "end-to-end behavior") to a specific framing about "specific resource…; evidence: Pulumi's own docs confirm property tests "run resource-level assertions" with access to "all input and output values of all cloud resources," while integration tests "deploy actual infrastructure to verify your code's behavior end-to-end."…; source: https://www.pulumi.com/what-is/how-to-step-up-cloud-infrastructure-testing/ (result 5, sentences 5-1, 5-16); https://www.pulumi.com/docs/iac/guides/testing/integration/ (result 6, sentence 6-4))
  • L95 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* 'Every service has logging enabled and shipped to the central log account.'" → ➖ not-a-claim (evidence: The text is a bullet point in the PR author's own documentation describing a desired infrastructure property ("Every service has logging enabled and shipped to the central log account"). It is not a falsifiable assertion about a third-part…; source: content/what-is/how-to-step-up-cloud-infrastructure-testing.md, L95)
  • L97 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "In Pulumi, property-level checks are usually written as policy as code using CrossGuard, and the same policy code runs against pulumi preview (blocking the m…" → ✅ verified (framing: strengthened — claim replaces "these checks" with "property-level checks", which is the correct referent from context; source's broader form proves the claim a…; evidence: The file at the property-testing section reads: "In Pulumi, these checks are usually written as policy as code using CrossGuard. The same policy code runs against pulumi preview (blocking the merge) and as a dep…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L101 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Integration tests deploy real cloud resources into an ephemeral environment, run end-to-end checks, then tear the environment down." → ✅ verified (evidence: The file at the relevant section reads: "Integration tests deploy real cloud resources into an ephemeral environment, run end-to-end checks, then tear the environment down." This is a verbatim match to the claim.; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L111 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Integration tests are the slowest and most expensive layer of infrastructure testing because they consume cloud resources." → ✅ verified (framing: strengthened — sources confirm integration tests are slower/more expensive than unit tests; the claim's "most expensive layer" framing is a superlative that ho…; evidence: Multiple sources confirm integration tests are slower and more expensive than other layers because they provision real cloud resources. The Pulumi page itself states "Integration tests deploy cloud resources and validate their actual behav…; source: https://www.pulumi.com/what-is/how-to-step-up-cloud-infrastructure-testing/ and https://spacelift.io/blog/infrastructure-as-code-iac-testing)
  • L113 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi has an integration testing guide at /docs/iac/using-pulumi/testing/integration/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL is not declared as an alias on content/docs/iac/guides/testing/integration/_index.md; canonical URL is /docs/iac/guides/testing/integration/]
  • L119 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Tools like Checkov, tfsec, Terrascan, and Snyk IaC scan Terraform, CloudFormation, and Pulumi outputs against a built-in ruleset." → ❌ contradicted (framing: narrowed — claim broadens Snyk IaC's supported frameworks to include Pulumi; source supports only Terraform, CloudFormation, Kubernetes, and ARM. Also, tfsec i…; evidence: (escalated from pass1) Snyk IaC's official docs state it supports "HashiCorp Terraform, AWS CloudFormation, Kubernetes, and Azure Resource Manager (ARM)" — Pulumi is not listed as a supported IaC scanning target. Additionally, tfsec was de…; source: https://docs.snyk.io/scan-with-snyk/snyk-iac)
  • L121 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi CrossGuard policies run during pulumi preview and pulumi up, so a non-compliant change can't get past CI." → ✅ verified (evidence: The file at content/what-is/how-to-step-up-cloud-infrastructure-testing.md contains the exact text: "Policies run during pulumi preview and pulumi up, so a non-compliant change can't get past CI." This matches the CrossGuard documentat…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L121 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi CrossGuard lets you author policies in TypeScript, Python, Java, or OPA's Rego against the actual Pulumi resource model." → ❌ contradicted (framing: shifted — claim adds "Java" as a supported CrossGuard policy language, but the source lists only TypeScript/JavaScript, Python, and OPA (Rego); Java is not men…; evidence: The official /docs/insights/policy/ page states: "Policies can be written in TypeScript/JavaScript (Node.js), Python, or OPA (Rego)" — Java is not a supported CrossGuard policy authoring language. The claim incorrectly includes Java as a…; source: repo:content/docs/insights/policy/_index.md)
  • L129 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Whatever the mix, the consistent rule is shift left: every security check that can run in CI should, and the deploy gate should fail closed." → ➖ not-a-claim (framing: The principle is well-supported by industry sources (e.g., CrowdStrike, Wiz, Spacelift), but the sentence is the PR author's own editorial guidance, not a cite…; evidence: The statement "shift left: every security check that can run in CI should, and the deploy gate should fail closed" is a general best-practice editorial assertion authored by the PR author themselves — it is not attributed to any third-part…; source: WebSearch ran query "shift left security CI fail closed deploy gate cloud infrastructure testing")
  • L146 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Unit testing for IaC uses Jest, Vitest, pytest, go test, xUnit, and JUnit as standard test runners for the language IaC is written in." → ✅ verified (evidence: The file at the unit testing section states: "Pulumi programs can be unit-tested using the standard test runners for the language you wrote them in (Jest/Vitest for TypeScript, pytest for Python, go test for Go, xUnit for C#, JUnit for J…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L147 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Checkov, tfsec, Terrascan, and Snyk IaC are representative tools for static IaC scanning." → ✅ verified (evidence: The file at L147 area explicitly states: "Tools like Checkov, tfsec, Terrascan, and Snyk IaC scan Terraform, CloudFormation, and Pulumi outputs against a built-in ruleset. Run them in CI on every change." — directly confirming these four t…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L148 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi CrossGuard, Open Policy Agent (OPA), and HashiCorp Sentinel are representative tools for policy as code." → ✅ verified (framing: strengthened — the source confirms CrossGuard and OPA; Sentinel is a real HashiCorp product not mentioned in the Pulumi docs but is a legitimate representative…; evidence: The /docs/insights/policy/ page confirms Pulumi CrossGuard and OPA (Rego) as policy-as-code tools. HashiCorp Sentinel is a well-known, real policy-as-code product from HashiCorp used with Terraform Cloud/Enterprise, making the claim fact…; source: repo:content/docs/insights/policy/_index.md)
  • L149 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi automation API, Terratest, and Kitchen-Terraform are representative tools for property and integration testing." → ✅ verified (framing: strengthened — the claim groups all three tools under "property and integration testing," but sources show Pulumi Automation API is specifically for integratio…; evidence: (escalated from pass1) Pulumi docs confirm the Automation API is used for integration testing; multiple sources confirm Terratest and Kitchen-Terraform are representative tools for Terraform integration testing. One source states: "Terrafo…; source: https://www.jit.io/blog/pulumi-vs-terraform-the-iac-showdown; https://www.pulumi.com/docs/using-pulumi/testing/; https://www.pulumi.com/docs/iac/guides/testing/integration/)
  • L150 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "LocalStack emulates AWS, Moto emulates AWS, and Azurite emulates Azure for cloud emulation in testing." → ✅ verified (framing: strengthened — Azurite specifically emulates Azure Storage services rather than all of Azure, so the claim is slightly broader than the most precise descriptio…; evidence: LocalStack is a well-known AWS cloud service emulator, Moto is a Python library that mocks/emulates AWS services, and Azurite is Microsoft's official Azure Storage emulator — all three are accurately attributed to their respective cloud pl…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md (file context) plus well-established public documentation for LocalStack, Moto, and Azurite)
  • L151 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Trivy, Snyk, Anchore, and Grype are representative tools for image and dependency scanning." → ✅ verified (framing: strengthened — the claim names "Anchore" alongside Grype; sources clarify Grype is Anchore's open-source scanner, so Anchore (as the maker/enterprise tier) and…; evidence: (escalated from pass1) Multiple authoritative sources confirm all four tools as representative image and dependency scanners. Grype is described as "a fast, open-source CLI vulnerability scanner built by Anchore" for container images; Triv…; source: WebSearch ran query "Trivy Snyk Anchore Grype container image dependency scanning tools"; confirmed by https://sesamedisk.com/container-security-scanning-trivy-grype-snyk/, https://aquilax.ai/blog/scan-docker-images-vulnerabilities, and https://secure-pipelines.com/ci-cd-security/ci-cd-security-scanners-compared-trivy-grype-snyk-checkov/)
  • L152 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Gremlin, AWS Fault Injection Simulator, and Chaos Mesh are representative tools for chaos engineering." → ✅ verified (evidence: (escalated from pass1) Multiple authoritative sources confirm all three tools as representative chaos engineering tools. Gremlin's own comparison page describes "AWS Fault Injection Simulator (FIS) lets you introduce faults into AWS servic…; source: https://www.gremlin.com/community/tutorials/chaos-engineering-tools-comparison)
  • L160 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi supports writing programs in TypeScript, Python, Go, C#, Java, or YAML." → ❌ contradicted (framing: shifted — claim lists "TypeScript, Python, Go, C#, Java, or YAML" but the authoritative source lists "TypeScript, JavaScript, Python, Go, .NET, Java, and YAML"…; evidence: The official Pulumi Languages & SDKs page states: "Pulumi supports TypeScript, JavaScript, Python, Go, .NET, Java, and YAML." The claim omits JavaScript and uses "C#" instead of ".NET" (the official designation, which covers C#, F#, and VB…; source: repo:content/docs/iac/languages-sdks/_index.md)
  • L161 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi's test mocks documentation is at /docs/iac/using-pulumi/testing/unit/." → ❌ contradicted (in-review link check: same URL as L84 — /docs/iac/using-pulumi/testing/unit/ is not declared as an alias on the unit-testing page; canonical URL is /docs/iac/guides/testing/unit/; source: repo:content/docs/iac/guides/testing/unit.md)
  • L161-164 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner, so integration tests can deploy and tear down ephemeral stacks p…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L162 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner, so integration tests can deploy and tear down ephemeral stacks p…" → ✅ verified (framing: strengthened — claim narrows the general "programmatic interface for driving Pulumi programs" to the specific test-runner/ephemeral-stack use case; the source'…; evidence: The Automation API README states it encapsulates "the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility" and explicitly lists "Integration testing" as a use case…; source: gh api repos/pulumi/pulumi/contents/sdk/go/auto/README.md (decoded base64 content))
  • L163 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "CrossGuard policies run during preview and update, blocking changes that violate organizational rules." → ✅ verified (evidence: The /docs/insights/policy/ page states under Enforcement modes: "Preventative: Validates Pulumi stack resources during pulumi preview and pulumi up, blocking deployments when violations are detected." This directly confirms CrossGuard…; source: content/docs/insights/policy/_index.md)
  • L164 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi runs in every major CI/CD platform via the GitHub Actions integration or any other system that can run a CLI." → 🤷 unverifiable (evidence: verification did not converge within 8 turns) [in-review link check confirms: URL /docs/iac/packages-and-automation/github-actions/ is not declared as an alias on content/docs/iac/guides/continuous-delivery/github-actions.md; canonical URL is /docs/iac/guides/continuous-delivery/github-actions/]
  • L166 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi supports provisioning and testing infrastructure as code in TypeScript, Python, Go, C#, Java, or YAML." → ❌ contradicted (framing: shifted — claim lists "TypeScript, Python, Go, C#, Java, or YAML" but the source lists "TypeScript, JavaScript, Python, Go, .NET, Java, and YAML"; JavaScript i…; evidence: The official Pulumi Languages & SDKs page states: "Pulumi supports TypeScript, JavaScript, Python, Go, .NET, Java, and YAML." The claim omits JavaScript entirely and uses "C#" instead of ".NET" (the broader SDK that covers C#, F#, and VB.N…; source: repo:content/docs/iac/languages-sdks/_index.md)
  • L176 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "IaC unit tests run in memory with mocked cloud responses and are fast (seconds), but only test the logic of IaC code." → ✅ verified (evidence: The document states: "Unit tests verify the logic of your IaC programs without calling a cloud provider. External dependencies are replaced with mocks that return canned responses, so the test runs entirely in memory. … Unit tests run in s…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md)
  • L184 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "In Pulumi, property tests are typically written as CrossGuard policies, and CrossGuard policy documentation is at /docs/insights/policy/." → ✅ verified (framing: strengthened — claim says "typically written as CrossGuard policies"; source says "usually written as policy as code using CrossGuard" — same meaning, claim is…; evidence: The source file at L184 states "In Pulumi, these checks are usually written as policy as code using CrossGuard," and the file content/docs/insights/policy/_index.md confirms the documentation exists at `/docs/in…; source: repo:content/what-is/how-to-step-up-cloud-infrastructure-testing.md and repo:content/docs/insights/policy/_index.md)
  • L204 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Compliance frameworks such as SOC 2, HIPAA, and PCI accept IaC test results as evidence, and increasingly they prefer it." → ❌ contradicted (framing: narrowed — claim broadens the source's "IaC test results can serve as compliance evidence" to "frameworks increasingly prefer them"; no authoritative source su…; evidence: (escalated from pass1) Sources confirm that SOC 2, HIPAA, and PCI DSS can accept IaC scan results as compliance evidence, but no authoritative source states these frameworks "increasingly prefer" IaC test results. One source notes "Auditor…; source: WebSearch ran query "SOC 2 HIPAA PCI DSS IaC infrastructure as code test results compliance evidence")
  • L204 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "A CrossGuard policy run produces a concrete artifact that proves a control was enforced on a specific change at a specific time, and compliance auditors prefer…" → 🤷 unverifiable (evidence: Official CrossGuard documentation describes policy enforcement, violation reporting, and compliance framework support, but no authoritative source describes a CrossGuard policy run as producing a "concrete artifact" proving enforcement at…; source: WebSearch ran query "Pulumi CrossGuard policy compliance audit artifact enforcement"; top results (pulumi.com/crossguard, pulumi.com/docs/iac/crossguard, github.com/pulumi/pulumi-policy) did not address the specific claim about auditor preferences or a concrete enforcement artifact.; intuition: The claim about auditor preferences ("compliance auditors prefer that to policy statements with no enforcement mechanis…)
  • L208 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Start with the cheapest layer that produces the most value: static scanning and policy as code in advisory mode. That gives you a baseline of how compliant the…" → ➖ not-a-claim (evidence: The text is a prescriptive methodology recommendation authored by the PR author ("Start with the cheapest layer…", "That gives you a baseline…", "Promote policies…", "Add unit tests…"). It contains no falsifiable assertion about a third-pa…; source: content/what-is/how-to-step-up-cloud-infrastructure-testing.md L208)
  • L212 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "Pulumi automation API documentation is at /docs/iac/packages-and-automation/automation-api/." → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L216-220 in content/what-is/how-to-step-up-cloud-infrastructure-testing.md "* What is Infrastructure as Code (IaC)?" → ✅ verified (evidence: The file content/what-is/what-is-infrastructure-as-code.md exists with title: "What is Infrastructure as Code (IaC)?", exactly matching the link text and path used in the claim.; source: repo:content/what-is/what-is-infrastructure-as-code.md)

📋 Triaged verifier findings

I double-checked these and realized they weren't real findings — click to expand

  • [L88] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Property tests run after a pulumi preview (or equivalent) produces a plan, against the planned resource graph or the freshly-deployed real resources." — Spurious: The verifier compared the claim against a partial source passage that emphasized only the during-deployment mode. The PR's framing actually covers both modes explicitly ("the planned resource graph or the freshly-deployed real resources"), which matches CrossGuard's documented behavior of running at both pulumi preview (against the plan) and pulumi up (against deployed resources).

  • [L160] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Write Pulumi programs in TypeScript, Python, Go, C#, Java, or YAML" — Spurious: The verifier strictly compared this to the canonical languages-SDK summary ("TypeScript, JavaScript, Python, Go, .NET, Java, and YAML") and flagged JavaScript-missing and "C#" vs ".NET". This 6-language shorthand (TS/Python/Go/C#/Java/YAML) is used widely throughout Pulumi's own marketing and docs prose — it's a stylistic convention, not a factual error. (A reader interested in JavaScript would not be misled, since TypeScript is the supported pathway for the Node.js runtime.)

  • [L166] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "to provision and test infrastructure as code in TypeScript, Python, Go, C#, Java, or YAML" — Spurious: Same shorthand as L160 in the closing CTA. Same reasoning applies.

  • [L161-164] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "The Pulumi automation API lets you script pulumi up and pulumi destroy from a test runner…" — Spurious: Verifier ran out of turns on a duplicate; the L162 entry for the same claim verified successfully against the Automation API README (which lists integration testing as a use case).

  • [L212] content/what-is/how-to-step-up-cloud-infrastructure-testing.md — "Pulumi automation API documentation is at /docs/iac/packages-and-automation/automation-api/." — Spurious: The URL resolves via alias. content/docs/iac/automation-api/_index.md declares /docs/iac/packages-and-automation/automation-api/ in its aliases: block, so this link works (canonical URL is /docs/iac/automation-api/ but the alias is intentional).

🚨 Outstanding in this PR

No outstanding findings.

⚠️ Low-confidence

No low-confidence findings.

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

  • [L84] Broken link to the unit testing guide — /docs/iac/using-pulumi/testing/unit/ corrected to /docs/iac/guides/testing/unit/ (resolved in 7d8d32c)

  • [L113] Broken link to the integration testing guide — /docs/iac/using-pulumi/testing/integration/ corrected to /docs/iac/guides/testing/integration/ (resolved in 7d8d32c)

  • [L119] Static-scanner sentence listed archived tfsec and claimed Snyk IaC scans Pulumi outputs (it doesn't) — replaced tfsec with Trivy throughout, narrowed format list to Terraform/CloudFormation/Kubernetes, added note that Checkov has a Pulumi-output mode (resolved in 7d8d32c)

  • [L121] CrossGuard policy authoring listed Java as a supported language (it isn't) — corrected to TypeScript/JavaScript, Python, OPA Rego; added parenthetical clarifying CrossGuard policies apply to stacks in any supported language including Go, .NET, and Java (resolved in 7d8d32c)

  • [L161] Broken link to test-mocks docs — /docs/iac/using-pulumi/testing/unit/ corrected to /docs/iac/guides/testing/unit/ (resolved in 7d8d32c)

  • [L164] Broken link to GitHub Actions integration — /docs/iac/packages-and-automation/github-actions/ corrected to /docs/iac/guides/continuous-delivery/github-actions/ (resolved in 7d8d32c)

  • [L204] "Yes, and increasingly they prefer it" / "Auditors much prefer that" framing overstated what compliance frameworks require — softened to "SOC 2, HIPAA, and PCI DSS audits routinely accept IaC test output and policy-as-code run logs as evidence that a control is enforced" (resolved in 7d8d32c)

  • [L204 ⚠️] Unverifiable "auditors much prefer" claim addressed by the same L204 rewrite (resolved in 7d8d32c)

📜 Review history

  • 2026-05-19T03:34:25Z — First review: 7 blockers (4 broken internal links to Pulumi guides, CrossGuard-Java mis-attribution, tfsec/Snyk-IaC tooling overreach, "increasingly prefer" framing overreach); 1 low-confidence note (09410e4)
  • 2026-05-19T05:12:00Z — re-reviewed after fix push by @alexleventer; all 7 blockers resolved, 1 low-confidence resolved (1 new commit, 7d8d32c)
  • 2026-05-19T17:49:08Z — author field updated to cam-soper per @CamSoper request; no content changes; 0 outstanding unchanged (1 new commit, eb3539f)

Need a re-review? Want to dispute a finding? Mention @claude and include #update-review.
(For ad-hoc questions or fixes, just @claude — no hashtag.)

@github-actions github-actions Bot added review:outstanding-issues Claude review completed; outstanding has author-actionable findings and removed review:in-progress Claude review is currently running labels May 19, 2026
@alexleventer
Copy link
Copy Markdown
Contributor Author

alexleventer commented May 19, 2026

@claude - fix all the outstanding items and then #update-review

@github-actions github-actions Bot added review:in-progress Claude review is currently running and removed review:outstanding-issues Claude review completed; outstanding has author-actionable findings labels May 19, 2026
@github-actions @claude
fix(what-is): address all outstanding review findings for cloud infra…
7d8d32c 
… testing explainer

- Fix 4 broken internal links (unit testing, integration testing, GitHub Actions)
- Remove archived tfsec scanner; replace with Trivy throughout
- Narrow Snyk IaC scanning claim to supported formats (removes Pulumi outputs)
- Fix CrossGuard policy authoring languages (remove Java, add JS clarification)
- Soften SOC 2/HIPAA/PCI compliance framing from "increasingly prefer" to "routinely accept"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

🤖 Review updated on @alexleventer's request.

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels May 19, 2026
CamSoper
CamSoper reviewed May 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@CamSoper CamSoper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude

  • replace the author with cam-soper
  • #update-review

@github-actions github-actions Bot added review:in-progress Claude review is currently running and removed review:no-blockers Claude review completed cleanly; outstanding is empty labels May 19, 2026
@github-actions @claude
fix(what-is): update author to cam-soper
eb3539f 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

🤖 Review updated on @CamSoper's request.

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels May 19, 2026
CamSoper
CamSoper reviewed May 19, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@CamSoper CamSoper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude Remove all references to CrossGuard (deprecated term) and then #update-review

@github-actions github-actions Bot added review:in-progress Claude review is currently running and removed review:no-blockers Claude review completed cleanly; outstanding is empty labels May 19, 2026
@github-actions @claude
fix(what-is): replace deprecated CrossGuard term with Pulumi Policies
8bc9017 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 19, 2026

🤖 Review updated on @CamSoper's request.

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels May 19, 2026
@pulumi-bot
Copy link
Copy Markdown
Collaborator

pulumi-bot commented May 19, 2026

Your site preview for commit 8bc9017 is ready! 🎉

http://www-testing-pulumi-docs-origin-pr-19146-8bc90177.s3-website.us-west-2.amazonaws.com

@CamSoper CamSoper enabled auto-merge (squash) May 19, 2026 19:39
@CamSoper CamSoper merged commit 65479d9 into master May 19, 2026
9 checks passed
@CamSoper CamSoper deleted the aleventer/what-is-cloud-infra-testing-rewrite branch May 19, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CamSoper CamSoper CamSoper approved these changes

Assignees

No one assigned

Labels

domain:docs PR touches technical docs review:no-blockers Claude review completed cleanly; outstanding is empty

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alexleventer @pulumi-bot @CamSoper