Scaling with the KEDA azure-pipelines scaler?

[guythetechie]

I can't get autoscaling to work with the KEDA azure-pipelines scaler. Here's my bicep configuration.

resource container_app 'Microsoft.Web/containerApps@2021-03-01' = {
  name: 'mycontainerapp'
  location: 'eastus'
  properties: {
    kubeEnvironmentId: container_app_environment.id
    configuration: {
      activeRevisionsMode: 'single'
      secrets: [
        {
          name: azure_devops_url_secret_name
          value: azure_devops_url
        }
        {
          name: azure_devops_pool_name_secret_name
          value: azure_devops_pool_name
        }
        {
          name: azure_devops_token_secret_name
          value: azure_devops_token
        }
        {
          name: container_registry_password_secret_name
          value: container_registry.listCredentials().passwords[0].value
        }
      ]
      registries: [
        {
          server: container_registry.properties.loginServer
          username: container_registry.listCredentials().username
          passwordSecretRef: container_registry_password_secret_name
        }
      ]
    }
    template: {
      containers: [
        {
          image: '${container_registry.properties.loginServer}/repo:123'
          name: 'mycontainerapp'
          env: [
            {
              name: 'AZP_URL'
              secretRef: azure_devops_url_secret_name
            }
            {
              name: 'AZP_TOKEN'
              secretRef: azure_devops_token_secret_name
            }
            {
              name: 'AZP_POOL'
              secretRef: azure_devops_pool_name_secret_name
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'azure-pipelines'
            custom: {
              type: 'azure-pipelines'
              metadata: {
                poolId: azure_devops_pool_id
                targetPipelinesQueueLength: '1'
              }
              auth: [
                {
                  triggerParameter: 'organizationURL'
                  secretRef: azure_devops_url_secret_name
                }
                {
                  triggerParameter: 'personalAccessToken'
                  secretRef: azure_devops_token_secret_name
                }
              ]
            }
          }
        ]
      }
    }
  }
}

The container deploys successfully with 1 replica. However, it doesn't scale no matter how many jobs are waiting in the Azure Pipelines queue.

[tomkerkhove]

Adding @SebastianSchuetze who is looking into this as well, I'll take a look from the KEDA side to see if we need to do something.

[SebastianSchuetze]

Same thing here and thanks @tomkerkhove for adding.

We have also no visibility if KEDA throws exceptions. Logs of Container Apps themselves during runtime should be surfaced as well.

[SebastianSchuetze]

For comparison I am adding my ARM template into the mix I used:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "defaultValue": "northeurope",
            "type": "String"
        },
        "environment_name": {
            "type": "String"
        },
        "azp_url": {
            "type": "String"
        },
        "azp_token": {
            "type": "String"
        },
        "azp_pool": {
            "type": "String"
        },
        "azp_poolId": {
            "type": "String"
        },
        "registryEndpoint": {
            "type": "String"
        },
        "registryUserName": {
            "type": "String"
        },
        "registrySecret": {
            "type": "String"
        },
        "registryImage": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [{
        "name": "azdo-agent-scaled",
        "type": "Microsoft.Web/containerApps",
        "apiVersion": "2021-03-01",
        "kind": "containerapp",
        "location": "[parameters('location')]",
        "properties": {
            "kubeEnvironmentId": "[resourceId('Microsoft.Web/kubeEnvironments', parameters('environment_name'))]",
            "configuration": {
                "secrets": [{
                        "name": "azp-url",
                        "value": "[parameters('azp_url')]"
                    },
                    {
                        "name": "azp-token",
                        "value": "[parameters('azp_token')]"
                    },
                    {
                        "name": "azp-pool",
                        "value": "[parameters('azp_pool')]"
                    },
                    {
                        "name": "azp-poolid",
                        "value": "[parameters('azp_poolId')]"
                    },
                    {
                        "name": "registry-secret",
                        "value": "[parameters('registrySecret')]"
                    }
                ],
                "registries": [{
                    "server": "[parameters('registryEndpoint')]",
                    "username": "[parameters('registryUserName')]",
                    "passwordSecretRef": "registry-secret"
                }]
            },
            "template": {
                "containers": [{
                    "image": "[parameters('registryImage')]",
                    "name": "basedockeragent",
                    "resources": {
                        "cpu": 0.75,
                        "memory": "1.5Gi"
                    },
                    "env": [{
                            "name": "AZP_URL",
                            "secretRef": "azp-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azp-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azp-pool"
                        },
                        {
                            "name": "AZP_POOLID",
                            "secretRef": "azp-poolid"
                        }
                    ]
                }],
                "scale": {
                    "minReplicas": "1",
                    "maxReplicas": "5",
                    "rules": [{
                        "name": "azdo-agent-scaled",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolID": "[parameters('azp_poolId')]",
                                "organizationURLFromEnv ": "[parameters('azp_url')]",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [{
                                "secretRef": "azp-token",
                                "triggerParameter": "personalAccessToken"
                            }]
                        }
                    }]
                }
            }
        }
    }]
}

[tomkerkhove]

@SebastianSchuetze You are providing the actual URL of your organization which should point to an environment variable.

What if you try this?

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "defaultValue": "northeurope",
            "type": "String"
        },
        "environment_name": {
            "type": "String"
        },
        "azp_url": {
            "type": "String"
        },
        "azp_token": {
            "type": "String"
        },
        "azp_pool": {
            "type": "String"
        },
        "azp_poolId": {
            "type": "String"
        },
        "registryEndpoint": {
            "type": "String"
        },
        "registryUserName": {
            "type": "String"
        },
        "registrySecret": {
            "type": "String"
        },
        "registryImage": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [{
        "name": "azdo-agent-scaled",
        "type": "Microsoft.Web/containerApps",
        "apiVersion": "2021-03-01",
        "kind": "containerapp",
        "location": "[parameters('location')]",
        "properties": {
            "kubeEnvironmentId": "[resourceId('Microsoft.Web/kubeEnvironments', parameters('environment_name'))]",
            "configuration": {
                "secrets": [{
                        "name": "azp-url",
                        "value": "[parameters('azp_url')]"
                    },
                    {
                        "name": "azp-token",
                        "value": "[parameters('azp_token')]"
                    },
                    {
                        "name": "azp-pool",
                        "value": "[parameters('azp_pool')]"
                    },
                    {
                        "name": "azp-poolid",
                        "value": "[parameters('azp_poolId')]"
                    },
                    {
                        "name": "registry-secret",
                        "value": "[parameters('registrySecret')]"
                    }
                ],
                "registries": [{
                    "server": "[parameters('registryEndpoint')]",
                    "username": "[parameters('registryUserName')]",
                    "passwordSecretRef": "registry-secret"
                }]
            },
            "template": {
                "containers": [{
                    "image": "[parameters('registryImage')]",
                    "name": "basedockeragent",
                    "resources": {
                        "cpu": 0.75,
                        "memory": "1.5Gi"
                    },
                    "env": [{
                            "name": "AZP_URL",
                            "secretRef": "azp-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azp-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azp-pool"
                        },
                        {
                            "name": "AZP_POOLID",
                            "secretRef": "azp-poolid"
                        }
                    ]
                }],
                "scale": {
                    "minReplicas": "1",
                    "maxReplicas": "5",
                    "rules": [{
                        "name": "azdo-agent-scaled",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolID": "[parameters('azp_poolId')]",
-                                "organizationURLFromEnv ": "[parameters('azp_url')]",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [{
                                "secretRef": "azp-token",
                                "triggerParameter": "personalAccessToken"
+                            },{
+                                "secretRef": "azp-url",
+                                "triggerParameter": "organizationURL"
                            }]
                        }
                    }]
                }
            }
        }
    }]
}

[SebastianSchuetze]

@tomkerkhove What is the difference? Isn't KEDA using this anyways the same way? Docs say mine is also valid from what I understood.

[tomkerkhove]

organizationURLFromEnv needs to be the name of the environment variable on your workload. But you specify the actual URL, instead of that name.

[SebastianSchuetze]

oh okay. Very good point!
Then it would be a really addition if KEDA / Dapr exceptions would be surfaced above the container as part of a new log type and everything around a container app.

[SebastianSchuetze]

@tomkerkhove it is f***ing working!!!

[SebastianSchuetze]

@guythetechie I don't see where you set the value for azure_devops_pool_id in your snippet.

[tomkerkhove]

Great to hear that it's working and the pool ID seems to be missing indeed!

[guythetechie]

@SebastianSchuetze - Glad you got it working. Now I know it's possible. :) It would definitely be useful to have KEDA/DAPR logs.

@tomkerkhove - I pass the pool ID as a parameter, but please let me know if I'm doing it wrong. Just to be clear though - the pool ID is a number (as opposed to a GUID), right? If my template looks good, next stop is making sure the values I'm passing are correct.

Here's the full Bicep file, parameters included:

param container_registry_name string
param container_app_environment_name string
param azure_devops_url string
@secure()
param azure_devops_token string
param azure_devops_pool_name string
param azure_devops_pool_id string

var azure_devops_url_secret_name = 'azure-devops-url'
var azure_devops_pool_name_secret_name = 'azure-devops-pool-name'
var azure_devops_token_secret_name = 'azure-devops-token'
var container_registry_password_secret_name = 'container-registry-password'

resource container_registry 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' existing = {
  name: container_registry_name
}

resource container_app_environment 'Microsoft.Web/kubeEnvironments@2021-02-01' existing = {
  name: container_app_environment_name
}

resource container_app 'Microsoft.Web/containerApps@2021-03-01' = {
  name: 'mycontainerapp'
  location: 'eastus'
  properties: {
    kubeEnvironmentId: container_app_environment.id
    configuration: {
      activeRevisionsMode: 'single'
      secrets: [
        {
          name: azure_devops_url_secret_name
          value: azure_devops_url
        }
        {
          name: azure_devops_pool_name_secret_name
          value: azure_devops_pool_name
        }
        {
          name: azure_devops_token_secret_name
          value: azure_devops_token
        }
        {
          name: container_registry_password_secret_name
          value: container_registry.listCredentials().passwords[0].value
        }
      ]
      registries: [
        {
          server: container_registry.properties.loginServer
          username: container_registry.listCredentials().username
          passwordSecretRef: container_registry_password_secret_name
        }
      ]
    }
    template: {
      containers: [
        {
          image: '${container_registry.properties.loginServer}/repo:123'
          name: 'mycontainerapp'
          env: [
            {
              name: 'AZP_URL'
              secretRef: azure_devops_url_secret_name
            }
            {
              name: 'AZP_TOKEN'
              secretRef: azure_devops_token_secret_name
            }
            {
              name: 'AZP_POOL'
              secretRef: azure_devops_pool_name_secret_name
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'azure-pipelines'
            custom: {
              type: 'azure-pipelines'
              metadata: {
                poolId: azure_devops_pool_id
                targetPipelinesQueueLength: '1'
              }
              auth: [
                {
                  triggerParameter: 'organizationURL'
                  secretRef: azure_devops_url_secret_name
                }
                {
                  triggerParameter: 'personalAccessToken'
                  secretRef: azure_devops_token_secret_name
                }
              ]
            }
          }
        ]
      }
    }
  }
}

And here's the deployed revision's resource JSON in Azure:

{
    "id": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Web/containerApps/***/revisions/***--azbpvln",
    "name": "***--azbpvln",
    "type": "Microsoft.Web/containerapps/revisions",
    "location": "eastus",
    "properties": {
        "createdTime": "2021-11-15T18:56:19+00:00",
        "fqdn": "",
        "template": {
            "containers": [
                {
                    "image": "***.azurecr.io/***:3873",
                    "name": "***",
                    "env": [
                        {
                            "name": "AZP_URL",
                            "secretRef": "azure-devops-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azure-devops-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azure-devops-pool-name"
                        }
                    ],
                    "resources": {
                        "cpu": 0.5,
                        "memory": "1Gi"
                    }
                }
            ],
            "scale": {
                "minReplicas": 1,
                "maxReplicas": 10,
                "rules": [
                    {
                        "name": "azure-pipelines",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolId": "13",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [
                                {
                                    "secretRef": "azure-devops-url",
                                    "triggerParameter": "organizationURL"
                                },
                                {
                                    "secretRef": "azure-devops-token",
                                    "triggerParameter": "personalAccessToken"
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "active": true,
        "replicas": 1,
        "trafficWeight": 0,
        "name": "***--azbpvln",
        "healthState": "Healthy",
        "provisioningState": "Provisioned"
    }
}

[guythetechie]

@SebastianSchuetze - If you don't mind, what permissions does your token have? Mine has:

and

[SebastianSchuetze]

For testing I used full scope. I will reduce the scoping later to see when something is not working. But in general "manage pool" should be enough. The PAR is only used for registering new agents. The agents themselves get short like ved tokens and authenticate themselves with their own tokens.

[guythetechie]

Thank you. I'll try full scope.

In addition to Read & manage* pool permissions, I thought the token might need permissions to read the pipelines queue so KEDA can autoscale.

[tomkerkhove]

@tomkerkhove - I pass the pool ID as a parameter, but please let me know if I'm doing it wrong. Just to be clear though - the pool ID is a number (as opposed to a GUID), right? If my template looks good, next stop is making sure the values I'm passing are correct.

The ID should be the pool ID of your Azure DevOps pool - It should not be used randomly. Probably this is your issue then?