Preserve rule IDs when updating rulesets

[cjolowicz]

This PR improves updating cloudflare_ruleset resources to preserve the IDs of unmodified rules. Currently, when we update a ruleset, every rule is created from scratch and all rule IDs change. We can avoid that by including the IDs of rules that haven't been modified in the request body.

The implementation uses ResourceData.GetChange to retrieve the old and new values of the ruleset. Each rule in the two versions of the ruleset is serialized to JSON. We build a map to associate the JSON of each old rule with its ID. The map allows us to look up the new rules and find their IDs if they didn't change.

A ruleset can have rules that are identical except for their ID. We exclude the ID from the JSON lookup key and allow lookup keys to be associated with multiple IDs. We preserve the order of these IDs to ensure that each copy of a rule keeps its original ID.

Note: This PR extracts two functions from a large code region, which makes the overall diff hard to read. I've separated these refactorings into refactor: commits to make the PR easier to review. The core changes are in the single feat: commit.

[github-actions]

changelog detected ✅

[jacobbednarz]

thanks for this one @cjolowicz 💯

all acceptance tests are passing.

TF_ACC=1 go test $(go list ./...) -v -run "^TestAccCloudflareRuleset_" -count 1 -parallel 1 -timeout 120m -parallel 1
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/consts	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/modifiers/defaults	[no test files]
testing: warning: no tests to run
PASS
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/provider	0.369s [no tests to run]
testing: warning: no tests to run
PASS
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/framework/service/example	0.532s [no tests to run]
=== RUN   TestAccCloudflareRuleset_WAFBasic
=== PAUSE TestAccCloudflareRuleset_WAFBasic
=== RUN   TestAccCloudflareRuleset_WAFManagedRuleset
=== PAUSE TestAccCloudflareRuleset_WAFManagedRuleset
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetOWASP
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetOWASP
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetOWASPBlockXSSWithAnomalyOver60
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetOWASPBlockXSSWithAnomalyOver60
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetOWASPOnlyPL1
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetOWASPOnlyPL1
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetDeployMultiple
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetDeployMultiple
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithSkip
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithSkip
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithTopSkipAndLastSkip
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithTopSkipAndLastSkip
=== RUN   TestAccCloudflareRuleset_SkipPhaseAndProducts
=== PAUSE TestAccCloudflareRuleset_SkipPhaseAndProducts
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetWithCategoryAndRuleBasedOverrides
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetWithCategoryAndRuleBasedOverrides
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetWithIDBasedOverrides
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetWithIDBasedOverrides
=== RUN   TestAccCloudflareRuleset_MagicTransitUpdateWithHigherPriority
    provider_test.go:184: Skipping acceptance test as 0da42c8d2132a9ddaf714f9e7c920711 is not configured for Magic Transit
--- SKIP: TestAccCloudflareRuleset_MagicTransitUpdateWithHigherPriority (0.00s)
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetWithPayloadLogging
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetWithPayloadLogging
=== RUN   TestAccCloudflareRuleset_RateLimit
=== PAUSE TestAccCloudflareRuleset_RateLimit
=== RUN   TestAccCloudflareRuleset_PreserveRuleIDs
=== PAUSE TestAccCloudflareRuleset_PreserveRuleIDs
=== RUN   TestAccCloudflareRuleset_CustomErrors
=== PAUSE TestAccCloudflareRuleset_CustomErrors
=== RUN   TestAccCloudflareRuleset_RequestOrigin
=== PAUSE TestAccCloudflareRuleset_RequestOrigin
=== RUN   TestAccCloudflareRuleset_TransformationRuleURIPath
=== PAUSE TestAccCloudflareRuleset_TransformationRuleURIPath
=== RUN   TestAccCloudflareRuleset_TransformationRuleURIQuery
=== PAUSE TestAccCloudflareRuleset_TransformationRuleURIQuery
=== RUN   TestAccCloudflareRuleset_TransformHTTPResponseHeaders
=== PAUSE TestAccCloudflareRuleset_TransformHTTPResponseHeaders
=== RUN   TestAccCloudflareRuleset_TransformationRuleURIPathAndQueryCombination
=== PAUSE TestAccCloudflareRuleset_TransformationRuleURIPathAndQueryCombination
=== RUN   TestAccCloudflareRuleset_TransformationRuleRequestHeaders
=== PAUSE TestAccCloudflareRuleset_TransformationRuleRequestHeaders
=== RUN   TestAccCloudflareRuleset_TransformationRuleResponseHeaders
=== PAUSE TestAccCloudflareRuleset_TransformationRuleResponseHeaders
=== RUN   TestAccCloudflareRuleset_ActionParametersMultipleSkips
=== PAUSE TestAccCloudflareRuleset_ActionParametersMultipleSkips
=== RUN   TestAccCloudflareRuleset_ActionParametersOverridesAction
=== PAUSE TestAccCloudflareRuleset_ActionParametersOverridesAction
=== RUN   TestAccCloudflareRuleset_ActionParametersHTTPDDoSOverride
=== PAUSE TestAccCloudflareRuleset_ActionParametersHTTPDDoSOverride
=== RUN   TestAccCloudflareRuleset_ActionParametersOverrideAllRulesetRules
=== PAUSE TestAccCloudflareRuleset_ActionParametersOverrideAllRulesetRules
=== RUN   TestAccCloudflareRuleset_AccountLevelCustomWAFRule
=== PAUSE TestAccCloudflareRuleset_AccountLevelCustomWAFRule
=== RUN   TestAccCloudflareRuleset_ExposedCredentialCheck
=== PAUSE TestAccCloudflareRuleset_ExposedCredentialCheck
=== RUN   TestAccCloudflareRuleset_Logging
=== PAUSE TestAccCloudflareRuleset_Logging
=== RUN   TestAccCloudflareRuleset_ConditionallySetActionParameterVersion
=== PAUSE TestAccCloudflareRuleset_ConditionallySetActionParameterVersion
=== RUN   TestAccCloudflareRuleset_WAFManagedRulesetWithActionManagedChallenge
=== PAUSE TestAccCloudflareRuleset_WAFManagedRulesetWithActionManagedChallenge
=== RUN   TestAccCloudflareRuleset_LogCustomField
=== PAUSE TestAccCloudflareRuleset_LogCustomField
=== RUN   TestAccCloudflareRuleset_ActionParametersOverridesThrashingStatus
=== PAUSE TestAccCloudflareRuleset_ActionParametersOverridesThrashingStatus
=== RUN   TestAccCloudflareRuleset_CacheSettings
=== PAUSE TestAccCloudflareRuleset_CacheSettings
=== RUN   TestAccCloudflareRuleset_Config
=== PAUSE TestAccCloudflareRuleset_Config
=== RUN   TestAccCloudflareRuleset_Redirect
=== PAUSE TestAccCloudflareRuleset_Redirect
=== RUN   TestAccCloudflareRuleset_DynamicRedirect
=== PAUSE TestAccCloudflareRuleset_DynamicRedirect
=== CONT  TestAccCloudflareRuleset_WAFBasic
--- PASS: TestAccCloudflareRuleset_WAFBasic (10.30s)
=== CONT  TestAccCloudflareRuleset_TransformationRuleURIPathAndQueryCombination
--- PASS: TestAccCloudflareRuleset_TransformationRuleURIPathAndQueryCombination (9.61s)
=== CONT  TestAccCloudflareRuleset_DynamicRedirect
--- PASS: TestAccCloudflareRuleset_DynamicRedirect (8.96s)
=== CONT  TestAccCloudflareRuleset_Redirect
--- PASS: TestAccCloudflareRuleset_Redirect (13.85s)
=== CONT  TestAccCloudflareRuleset_Config
--- PASS: TestAccCloudflareRuleset_Config (8.99s)
=== CONT  TestAccCloudflareRuleset_CacheSettings
--- PASS: TestAccCloudflareRuleset_CacheSettings (36.42s)
=== CONT  TestAccCloudflareRuleset_ActionParametersOverridesThrashingStatus
--- PASS: TestAccCloudflareRuleset_ActionParametersOverridesThrashingStatus (49.52s)
=== CONT  TestAccCloudflareRuleset_LogCustomField
--- PASS: TestAccCloudflareRuleset_LogCustomField (9.37s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetWithActionManagedChallenge
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetWithActionManagedChallenge (17.41s)
=== CONT  TestAccCloudflareRuleset_ConditionallySetActionParameterVersion
--- PASS: TestAccCloudflareRuleset_ConditionallySetActionParameterVersion (17.81s)
=== CONT  TestAccCloudflareRuleset_Logging
=== CONT  TestAccCloudflareRuleset_ExposedCredentialCheck
--- PASS: TestAccCloudflareRuleset_Logging (10.22s)
--- PASS: TestAccCloudflareRuleset_ExposedCredentialCheck (10.09s)
=== CONT  TestAccCloudflareRuleset_AccountLevelCustomWAFRule
--- PASS: TestAccCloudflareRuleset_AccountLevelCustomWAFRule (13.83s)
=== CONT  TestAccCloudflareRuleset_ActionParametersOverrideAllRulesetRules
--- PASS: TestAccCloudflareRuleset_ActionParametersOverrideAllRulesetRules (8.96s)
=== CONT  TestAccCloudflareRuleset_ActionParametersHTTPDDoSOverride
--- PASS: TestAccCloudflareRuleset_ActionParametersHTTPDDoSOverride (10.56s)
=== CONT  TestAccCloudflareRuleset_ActionParametersOverridesAction
--- PASS: TestAccCloudflareRuleset_ActionParametersOverridesAction (9.45s)
=== CONT  TestAccCloudflareRuleset_TransformationRuleRequestHeaders
--- PASS: TestAccCloudflareRuleset_TransformationRuleRequestHeaders (9.30s)
=== CONT  TestAccCloudflareRuleset_ActionParametersMultipleSkips
--- PASS: TestAccCloudflareRuleset_ActionParametersMultipleSkips (10.89s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetWithIDBasedOverrides
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetWithIDBasedOverrides (10.12s)
=== CONT  TestAccCloudflareRuleset_TransformHTTPResponseHeaders
--- PASS: TestAccCloudflareRuleset_TransformHTTPResponseHeaders (11.07s)
=== CONT  TestAccCloudflareRuleset_TransformationRuleURIQuery
--- PASS: TestAccCloudflareRuleset_TransformationRuleURIQuery (10.09s)
=== CONT  TestAccCloudflareRuleset_TransformationRuleURIPath
--- PASS: TestAccCloudflareRuleset_TransformationRuleURIPath (9.45s)
=== CONT  TestAccCloudflareRuleset_RequestOrigin
--- PASS: TestAccCloudflareRuleset_RequestOrigin (9.74s)
=== CONT  TestAccCloudflareRuleset_CustomErrors
--- PASS: TestAccCloudflareRuleset_CustomErrors (9.88s)
=== CONT  TestAccCloudflareRuleset_PreserveRuleIDs
--- PASS: TestAccCloudflareRuleset_PreserveRuleIDs (37.32s)
=== CONT  TestAccCloudflareRuleset_RateLimit
--- PASS: TestAccCloudflareRuleset_RateLimit (8.83s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetWithPayloadLogging
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetWithPayloadLogging (9.15s)
=== CONT  TestAccCloudflareRuleset_TransformationRuleResponseHeaders
--- PASS: TestAccCloudflareRuleset_TransformationRuleResponseHeaders (9.02s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetOWASPBlockXSSWithAnomalyOver60
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetOWASPBlockXSSWithAnomalyOver60 (9.32s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetOWASPOnlyPL1
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetOWASPOnlyPL1 (9.18s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetDeployMultiple
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetDeployMultiple (9.80s)
=== CONT  TestAccCloudflareRuleset_SkipPhaseAndProducts
--- PASS: TestAccCloudflareRuleset_SkipPhaseAndProducts (9.03s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithTopSkipAndLastSkip
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithTopSkipAndLastSkip (9.65s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetWithCategoryAndRuleBasedOverrides
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetWithCategoryAndRuleBasedOverrides (9.18s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithSkip
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetDeployMultipleWithSkip (9.39s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRulesetOWASP
--- PASS: TestAccCloudflareRuleset_WAFManagedRulesetOWASP (8.99s)
=== CONT  TestAccCloudflareRuleset_WAFManagedRuleset
--- PASS: TestAccCloudflareRuleset_WAFManagedRuleset (8.89s)
PASS
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/sdkv2provider	474.096s
?   	github.com/cloudflare/terraform-provider-cloudflare/internal/utils	[no test files]

[cjolowicz]

thanks @jacobbednarz 🚀

[github-actions]

This functionality has been released in v3.34.0 of the Terraform Cloudflare Provider.

Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!