Support credentials objects for user-assigned Managed Identities

[kiriasCMNH]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
We were trying to use the new support that was added recently by microsoft to allow to use user-assigned managed identities for credentials for some data factory objects such as linked services for example, but when we tried to deploy that version that contained an autogenerated credential folder containing the related user assigned identity, the cmdlet threw an error inside the AdfObject.class.ps1 ADFT0029: Unknown object type: Credential.
Basically our linked service had a credential property inside its TypeProperties using a type CredentialReference

Describe the solution you'd like
A clear and concise description of what you want to happen.
It would be nice if that could be supported somehow.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
For now because of this restriction we were planning on still using plain connection strings with user and password for now, which we already handle through the csv substitution.

Additional context
Add any other context or screenshots about the feature request here.

[NowinskiK]

Currently, Credentials are not supported by Az.DataFactory module. So, it is blocked by this.

[dataengine01]

Hi

are there any updates on this? do we know when DataFactory module will include credentials and userAssignedIdentites?

[NowinskiK]

Mentioned: https://twitter.com/NowinskiK/status/1502616575899324417

[LeMaciek]

Hi,
It seems like the az module supports it now, I haven't tested the update part, but the needed parameters seem to be there. It would be nice to have support for managed identities in this module as well.

[Viguro79]

Hi. Same issue here. Can I help to fix it? Trying to understand... Should we only add 'credential' to $AllowedTypes in private\AdfObject.class.ps1 file to make it work or is there something else to change?

[NowinskiK]

No, az.datafactory PS module still doesn't have the appropriate cmdlet to support this type of object.
I raised that to Microsoft yesterday.
Thanks for offering your help though, @Viguro79

[Viguro79]

Ok. Thanks. While waiting for an update on the Azure side, would it be possible to add an exception in the module to skip the credentials deployment? We deploy them at creation of the DataFactory with Terraform :


Something like :

if 'type' == 'credential'
   then # do nothing on this object
else validate_object() and deploy_object()

[Viguro79]

We have added the following line in the code to skip credentials objets :
In /azure.datafactory.tools/private/Get-ReferencedObjects.ps1, after line 11, add this condition : if ($type.Equals('CredentialReference')) { $script:ind--; return }

    -and $node.PSobject.Properties.Name -contains 'type') 
{
    [string] $type = $node.type
    if ($type.Equals('CredentialReference')) { $script:ind--; return }
    if ($type.EndsWith('Reference')) {
        $type = $type.Substring(0, $type.Length-9)
        #Write-Verbose "$type.$($node.referenceName)"

So that the module won't stop anymore on ADFT0029: Unknown object type: Credential error.
However, we are in a situation where we already manage credentials deployments via Terraform. So that the targeted DataFactory is still able to find a valid credential when deploying.

[tommydejong]

Hi @Viguro79, can you share some insights on how you manage credentials with Terraform? There doesn't seem to be a TF resource for the credentials object yet either. We are in a similar situation but unfortunately not in a position where we can make changes to the Powershell module, so this issue is currently blocking us from deploying ADF objects. If it were possible to do it with Terraform I could work with the azurerm_data_factory_* resources but it would still need to deploy credential objects.

[NowinskiK]

Sorry guys for the delay. This gonna be implemented today.

[NowinskiK]

Currently deployed on preview version:
https://www.powershellgallery.com/packages/azure.datafactory.tools/0.98.0-preview
Please check, test and let me know if that works as expected.

[Viguro79]

Hi @Viguro79, can you share some insights on how you manage credentials with Terraform?

We are using an ARM template to deploy it :

This is the arm template we use :

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "type": "string",
            "metadata": "Data Factory name"
        },
        "identId": {
            "type": "string",
            "metadata": "the managed identiry id"
        },
        "credName": {
            "type": "string",
            "metadata": "Credential name"
        }
    },
    "variables": {
        "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
    },
    "resources": [
        {
            "name": "[concat(parameters('factoryName'), '/',parameters('credName'))]",
            "type": "Microsoft.DataFactory/factories/credentials",
            "apiVersion": "2018-06-01",
            "properties": {
                "type": "ManagedIdentity",
                "typeProperties": {
                    "resourceId": "[parameters('identId')]"
                }
            },
            "dependsOn": []
        }
    ]
}

And the terraform we use to deploy it in live mode :

resource "azurerm_resource_group_template_deployment" "rgp_cred" {
  name = "Credential_RGP"
  resource_group_name = var.rgp_name
  deployment_mode = "Incremental"
  template_content = file("${var.MODULE_FACTORY_DIR}/modules/datafactory/cred_arm-template/arm_template.json")
  # template_content = file("../datafactory/cred_arm-template/arm_template.json")
  parameters_content = jsonencode ({
    factoryName = {value = azurerm_data_factory.current.name}
    identId = {value = var.mgd_id_rgp}
    credName = {value = "cred-prod"}
  })
  depends_on = [azurerm_data_factory.current]
}

[Viguro79]

Currently deployed on preview version: https://www.powershellgallery.com/packages/azure.datafactory.tools/0.98.0-preview Please check, test and let me know if that works as expected.

Better. But still an issue there :

STEP: Deployment of all ADF objects...
Start deploying object: [linkedService].[ls_adls_projectname] (2 dependency/ies)
Write-Error: /home/AzDevOps/.local/share/powershell/Modules/azure.datafactory.tools/0.98.0/private/Deploy-AdfObject.ps1:29
Line |
  29 |  …   $depobj = Get-AdfObjectByName -adf $adf -name "$name" -type "$type" …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | ADFT0014: Type [Credential] is not supported.

[NowinskiK]

Check the latest:
https://www.powershellgallery.com/packages/azure.datafactory.tools/0.99.0-preview

[Viguro79]

This one is working as excepted.

[NowinskiK]

Released in ver.0.99