Error related to global_parameter for azurerm_data_factory resource, post provider upgrade to v2.64.0

[andy-mcgrath]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/azuread v1.5.1
+ provider registry.terraform.io/hashicorp/azurerm v2.64.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

Also tested using Terrarom 15.0.5 and 1.0.0 with the same providers, getting the same error.

Affected Resource(s)

• azurerm_data_factory

Terraform Configuration Files

This is a extract from a module named environment

# versions.tf
terraform {
  required_version = ">= 0.14"
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "=2.64.0"
    }
  }
}

provider "azurerm" {
  features {}
}

# main.tf
# Resource group
resource "azurerm_resource_group" "resource_group" {
    name     = "Data-Platform-Test"
    location = "UK South"
}

# Data factory
resource "azurerm_data_factory" "data_factory" {
  name                = "company-test-data-factory"
  resource_group_name = azurerm_resource_group.resource_group.name
  location            = azurerm_resource_group.resource_group.location

  global_parameter {
    name = "tfint"
    type   = "Int"
    value = 5
  }

  identity {
    type = "SystemAssigned"
  }
}

Debug Output

Debug output for command terraform plan -target module.environment.azurerm_data_factory.data_factory to reduce size

azurerm 2.64.0 azurerm_data_factory debug output

Expected Behaviour

Only change was upgrade the azurerm provider so expected no changes.

Actual Behaviour

In terraform plan replorting the following error or similar (noted the index value in global_parameter.0.value differ with each execution):

Error: setting `global_parameter`: global_parameter.0.value: '' expected type 'string', got unconvertible type 'float64'

Steps to Reproduce

terraform init
terraform plan
terraform apply

After creating the Azure Data Factory manually add a Global Parameter of type Bool, Int, Float or Array and publish

Finally run terraform plan to generate the error

Important Factoids

none

References

• azurerm_data_factory - support for global_parameter #12178

[favoretti]

@andy-mcgrath Hi there and thank you for reporting this. What puzzles me is that if I look at the actual response of the API I see this:

{
  "name": "comp-staging-data-factory",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/data-platform/providers/Microsoft.DataFactory/factories/comp-staging-data-factory",
  "type": "Microsoft.DataFactory/factories",
  "properties": {
    "provisioningState": "Succeeded",
    "createTime": "2020-08-28T14:17:50.4955238Z",
    "version": "2018-06-01",
    "factoryStatistics": {
      "totalResourceCount": 0,
      "maxAllowedResourceCount": 0,
      "factorySizeInGbUnits": 0,
      "maxAllowedFactorySizeInGbUnits": 0
    },
    "globalParameters": {
      "FactorySubscriptionId": {
        "type": "string",
        "value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      },
      "FactoryResourceGroup": {
        "type": "string",
        "value": "Data-platform"
      },
      "PowerBiSharedDatasetWorkspace": {
        "type": "string",
        "value": "SharedDatasets-Staging"
      },
      "LinkedInOrganizationEntityUrn": {
        "type": "string",
        "value": "urn%3Ali%3Aorganization%3A66616100"
      },
      "OneDayInSeconds": {
        "type": "int",
        "value": 86400
      },
      "SharedKeyVaultSecretsURI": {
        "type": "string",
        "value": "https://kv-comp-dp-shared.vault.azure.net/secrets/<secret-name>?api-version=7.1"
      },
      "CosmosAccountUriTraining": {
        "type": "string",
        "value": "https://xxxx.azure.com:443/"
      },
      "CosmosAccountUriProduction": {
        "type": "string",
        "value": "https://xxxx.documents.azure.com:443/"
      },
      "EnvironmentName": {
        "type": "string",
        "value": "Staging"
      }
    }
  },

I.e. there are no float64 parameters. Also, looking at the code we don't seem to do any conversions and it seems to work fine with my ADF instances... a bit puzzled here.

[andy-mcgrath]

Managed to replicate the issue. If I create a new Data Factory and use Terraform to create the global parameters the response looks like the following:

"globalParameters":{
  "tfint":{
    "type":"Int",
    "value":"5"
  }
}

However, if the Data Factory has existing global parameters or I add one manually via the Azure portal the response looks like the type and value get updated for both the new parameter and the one created by Terraform (both generate the error on terraform plan):

"globalParameters":{
  "tfint":{
    "type":"Int",
    "value":5
  },
  "manualint":{
    "type":"int",
    "value":5
  }
}

I suspect that any Data Factory with existing global parameters that at of type Bool, Int, Float or Array will generate this error if they upgrade to azurerm v2.64.0.

A possible fix would be to remove all global parameters and add them to Terraform but not able to test currently as I need input from another team internally.

[favoretti]

Thank you for your debugging effort! Interesting. This would be hard to reproduce from an acceptance test, since it involves meddling with a resource outside of terraform. I'll try to debug this further in the next couple of days if noone else beats me to it.

[cinni99]

Just would like to confirm , that I have a similar issue.
Data Factory has been deployed time ago without providing the Global Parameters by TF.
Now after an Provider Upgrade to v2.66.0 from 2.57.0 I now have the error:
"Error: setting global_parameter: global_parameter.0.value: '' expected type 'string', got unconvertible type '[]interface {}', value:...
Global Parameter has beed defined as "Array" in ADF Portal.

[7even7]

This problem still occurs in azurerm 2.73.0.
We had previously created parameters with type String. I have added them to our TF-configuration, but now plan&apply will always recreate them causing data factorys publish to see all related pipes as modified.

[andy-mcgrath]

I have been looking at this today (the first chance I have had) and identify two issues.

• Types look to be case-insensitive through the API docs use title style e.g. Array but updates made by the GUI use lowercase only.
• Values can be strings or the direct type via the API, example for a type Int both 3 and "3" are valid. If updating or adding via the GUI uses the correct type for the Value.

My thinking is to do the following:

• Update the Type to ignore case differences
• On read convert all types to strings. Likely need to check the global-parameter Type so can use an appropriate conversation option. Boolean, string, int and float are simple and maybe JSON encode for Array and Object types.

Not sure this is the best solution long term but should solve the majority of issues without being a breaking change.

If I get time this weekend will have a PR ready next week.

[frolovandreydanone]

Hello, looks like this bug is also affecting deployment when we set ignore_changes lifecycle block for global_parameter:

  lifecycle {
    ignore_changes = [
      global_parameter,
    ]
  }

Error I got:
Error: setting global_parameter: global_parameter.1.value: '' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[myKey:3]'

ADF:

This applies to both Array and Object types of parameters.

[andy-mcgrath]

The error is due to the type returned by the API being different than the schema type.

I have an open PR to fix the error, just waiting on a review.

[choukimasri]

This error is still present in 2.81, @favoretti @katbyte , can you please review the pr proposed above by @andy-mcgrath ??

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.