[K9VULN-14660] fix(agentless-azure): harden agentless azure setup

[mohamed-challal]

Summary

Hardens the Azure Agentless Scanner setup against the failure modes we expect to hit once it leaves pre-release: re-runs from a fresh Cloud Shell, repeated deploys with different SCANNER_RESOURCE_GROUP values, partial cleanups that leave Azure soft-deleted resources behind, and delegated-permission enterprise environments. Also roughly halves first-deploy time by parallelising the slowest phases.

The plumbing change underneath is a new per-install identifier (install-id = sha256(scanner_sub | rg)[:12]) used to derive Storage Account and Key Vault names, plus an Azure-tag-based discovery mechanism (DatadogAgentlessScanner=true) that replaces local state as the source of truth for "is there already a deployment in this subscription?".

Changes by theme

Robustness for re-runs and edge cases

• fix(agentless-azure): handle unpurged secret key during re-setup with a different RG recover Azure soft-deleted Key Vaults instead of crashing with ConflictError on re-deploy.
• fix(agentless-azure): handle resource group mismatch on deploy/destroy fail loudly (no silent overwrite) with actionable guidance when the user changes SCANNER_RESOURCE_GROUP between runs.
• agentless azure: derive install_id and discover existing deployments via RG tag tag-based RG discovery on both deploy and destroy:
  • 1 tagged RG, env var unset → silently adopt it (fresh Cloud Shell re-runs work without env var)
  • 1 tagged RG, env var disagrees → ConfigurationError with mismatch guidance
  • ≥2 tagged RGs → single-install policy: deploy refuses, destroy requires explicit env var
• agentless azure: scope storage account and key vault names to install_id resource names now derive from (scanner_sub, rg), so two RGs in the same subscription will not collide on a future multi-install iteration. Drops the legacy SA-RG lookup fallback.

Performance

First-deploy wall-clock on Cloud Shell drops roughly in half.

• perf(agentless-azure): parallelize lookup checks and resource creation Storage Account and Key Vault control-plane work runs concurrently.
• perf(agentless-azure): run preflight checks in parallel
• perf(agentless-azure): bump Terraform parallelism 10 -> 20
• fix(agentless-azure): skip Key Vault secret retries when Secrets Officer already exists

Permissions / enterprise delegation

• add(agentless-azure): make permissions check softer in preflight if RG already exists when the RG is pre-created (admin-provisioned enterprise pattern), the resource-creation actions are probed at RG scope rather than subscription scope, so engineers with RG-only Contributor pass preflight.
• add(agentless-azure): add roleDefinitions/write permission in preflight the Terraform roles module creates a custom scanning role whose assignableScopes covers every scan-target; surfacing this in preflight turns a confusing mid-apply 403 into an actionable error at the start of the run.

Observability

• fix(agentless): mark active workflow step FAILED when Azure deploy exits with error failures now flip the in-progress step to FAILED on the workflow-status API; the Datadog UI setup-progress timeline no longer spins forever when, e.g., terraform apply exits with insufficient permissions.

Cleanup / housekeeping

• refactor(agentless-azure): remove dead code
• docs(agentless-azure): update readme
• build(agentless-azure): re-build dist scripts refreshed dist/azure_agentless_setup.pyz.

Risk & compatibility

• Single-install policy is now enforced. Any subscription that somehow ended up with two tagged Agentless RGs will fail deploy fast (this is the intended behaviour — the alternative would be a silent pick).
• Tag contract. Discovery relies on the DatadogAgentlessScanner=true tag the script and Terraform module already apply on resource creation. The script never re-tags resources it didn't create (admin-pre-created RGs stay untagged and require the env var on subsequent re-runs).

[mohamed-challal]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4657b04b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tedkahwaji]

LGTM, mind updating the code owners so the gcp/azure agentless scripts tag your team?

[BraisCaboFelpete]

LGTM