Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scaling with the KEDA azure-pipelines scaler? #31

Closed
guythetechie opened this issue Nov 16, 2021 · 28 comments
Closed

Scaling with the KEDA azure-pipelines scaler? #31

guythetechie opened this issue Nov 16, 2021 · 28 comments

Comments

@guythetechie
Copy link

I can't get autoscaling to work with the KEDA azure-pipelines scaler. Here's my bicep configuration.

resource container_app 'Microsoft.Web/containerApps@2021-03-01' = {
  name: 'mycontainerapp'
  location: 'eastus'
  properties: {
    kubeEnvironmentId: container_app_environment.id
    configuration: {
      activeRevisionsMode: 'single'
      secrets: [
        {
          name: azure_devops_url_secret_name
          value: azure_devops_url
        }
        {
          name: azure_devops_pool_name_secret_name
          value: azure_devops_pool_name
        }
        {
          name: azure_devops_token_secret_name
          value: azure_devops_token
        }
        {
          name: container_registry_password_secret_name
          value: container_registry.listCredentials().passwords[0].value
        }
      ]
      registries: [
        {
          server: container_registry.properties.loginServer
          username: container_registry.listCredentials().username
          passwordSecretRef: container_registry_password_secret_name
        }
      ]
    }
    template: {
      containers: [
        {
          image: '${container_registry.properties.loginServer}/repo:123'
          name: 'mycontainerapp'
          env: [
            {
              name: 'AZP_URL'
              secretRef: azure_devops_url_secret_name
            }
            {
              name: 'AZP_TOKEN'
              secretRef: azure_devops_token_secret_name
            }
            {
              name: 'AZP_POOL'
              secretRef: azure_devops_pool_name_secret_name
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'azure-pipelines'
            custom: {
              type: 'azure-pipelines'
              metadata: {
                poolId: azure_devops_pool_id
                targetPipelinesQueueLength: '1'
              }
              auth: [
                {
                  triggerParameter: 'organizationURL'
                  secretRef: azure_devops_url_secret_name
                }
                {
                  triggerParameter: 'personalAccessToken'
                  secretRef: azure_devops_token_secret_name
                }
              ]
            }
          }
        ]
      }
    }
  }
}

The container deploys successfully with 1 replica. However, it doesn't scale no matter how many jobs are waiting in the Azure Pipelines queue.

@tomkerkhove
Copy link
Member

Adding @SebastianSchuetze who is looking into this as well, I'll take a look from the KEDA side to see if we need to do something.

@SebastianSchuetze
Copy link

Same thing here and thanks @tomkerkhove for adding.

We have also no visibility if KEDA throws exceptions. Logs of Container Apps themselves during runtime should be surfaced as well.

@SebastianSchuetze
Copy link

For comparison I am adding my ARM template into the mix I used:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "defaultValue": "northeurope",
            "type": "String"
        },
        "environment_name": {
            "type": "String"
        },
        "azp_url": {
            "type": "String"
        },
        "azp_token": {
            "type": "String"
        },
        "azp_pool": {
            "type": "String"
        },
        "azp_poolId": {
            "type": "String"
        },
        "registryEndpoint": {
            "type": "String"
        },
        "registryUserName": {
            "type": "String"
        },
        "registrySecret": {
            "type": "String"
        },
        "registryImage": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [{
        "name": "azdo-agent-scaled",
        "type": "Microsoft.Web/containerApps",
        "apiVersion": "2021-03-01",
        "kind": "containerapp",
        "location": "[parameters('location')]",
        "properties": {
            "kubeEnvironmentId": "[resourceId('Microsoft.Web/kubeEnvironments', parameters('environment_name'))]",
            "configuration": {
                "secrets": [{
                        "name": "azp-url",
                        "value": "[parameters('azp_url')]"
                    },
                    {
                        "name": "azp-token",
                        "value": "[parameters('azp_token')]"
                    },
                    {
                        "name": "azp-pool",
                        "value": "[parameters('azp_pool')]"
                    },
                    {
                        "name": "azp-poolid",
                        "value": "[parameters('azp_poolId')]"
                    },
                    {
                        "name": "registry-secret",
                        "value": "[parameters('registrySecret')]"
                    }
                ],
                "registries": [{
                    "server": "[parameters('registryEndpoint')]",
                    "username": "[parameters('registryUserName')]",
                    "passwordSecretRef": "registry-secret"
                }]
            },
            "template": {
                "containers": [{
                    "image": "[parameters('registryImage')]",
                    "name": "basedockeragent",
                    "resources": {
                        "cpu": 0.75,
                        "memory": "1.5Gi"
                    },
                    "env": [{
                            "name": "AZP_URL",
                            "secretRef": "azp-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azp-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azp-pool"
                        },
                        {
                            "name": "AZP_POOLID",
                            "secretRef": "azp-poolid"
                        }
                    ]
                }],
                "scale": {
                    "minReplicas": "1",
                    "maxReplicas": "5",
                    "rules": [{
                        "name": "azdo-agent-scaled",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolID": "[parameters('azp_poolId')]",
                                "organizationURLFromEnv ": "[parameters('azp_url')]",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [{
                                "secretRef": "azp-token",
                                "triggerParameter": "personalAccessToken"
                            }]
                        }
                    }]
                }
            }
        }
    }]
}

@tomkerkhove
Copy link
Member

@SebastianSchuetze You are providing the actual URL of your organization which should point to an environment variable.

What if you try this?

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "defaultValue": "northeurope",
            "type": "String"
        },
        "environment_name": {
            "type": "String"
        },
        "azp_url": {
            "type": "String"
        },
        "azp_token": {
            "type": "String"
        },
        "azp_pool": {
            "type": "String"
        },
        "azp_poolId": {
            "type": "String"
        },
        "registryEndpoint": {
            "type": "String"
        },
        "registryUserName": {
            "type": "String"
        },
        "registrySecret": {
            "type": "String"
        },
        "registryImage": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [{
        "name": "azdo-agent-scaled",
        "type": "Microsoft.Web/containerApps",
        "apiVersion": "2021-03-01",
        "kind": "containerapp",
        "location": "[parameters('location')]",
        "properties": {
            "kubeEnvironmentId": "[resourceId('Microsoft.Web/kubeEnvironments', parameters('environment_name'))]",
            "configuration": {
                "secrets": [{
                        "name": "azp-url",
                        "value": "[parameters('azp_url')]"
                    },
                    {
                        "name": "azp-token",
                        "value": "[parameters('azp_token')]"
                    },
                    {
                        "name": "azp-pool",
                        "value": "[parameters('azp_pool')]"
                    },
                    {
                        "name": "azp-poolid",
                        "value": "[parameters('azp_poolId')]"
                    },
                    {
                        "name": "registry-secret",
                        "value": "[parameters('registrySecret')]"
                    }
                ],
                "registries": [{
                    "server": "[parameters('registryEndpoint')]",
                    "username": "[parameters('registryUserName')]",
                    "passwordSecretRef": "registry-secret"
                }]
            },
            "template": {
                "containers": [{
                    "image": "[parameters('registryImage')]",
                    "name": "basedockeragent",
                    "resources": {
                        "cpu": 0.75,
                        "memory": "1.5Gi"
                    },
                    "env": [{
                            "name": "AZP_URL",
                            "secretRef": "azp-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azp-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azp-pool"
                        },
                        {
                            "name": "AZP_POOLID",
                            "secretRef": "azp-poolid"
                        }
                    ]
                }],
                "scale": {
                    "minReplicas": "1",
                    "maxReplicas": "5",
                    "rules": [{
                        "name": "azdo-agent-scaled",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolID": "[parameters('azp_poolId')]",
-                                "organizationURLFromEnv ": "[parameters('azp_url')]",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [{
                                "secretRef": "azp-token",
                                "triggerParameter": "personalAccessToken"
+                            },{
+                                "secretRef": "azp-url",
+                                "triggerParameter": "organizationURL"
                            }]
                        }
                    }]
                }
            }
        }
    }]
}

@SebastianSchuetze
Copy link

SebastianSchuetze commented Nov 16, 2021

@tomkerkhove What is the difference? Isn't KEDA using this anyways the same way? Docs say mine is also valid from what I understood.

@tomkerkhove
Copy link
Member

organizationURLFromEnv needs to be the name of the environment variable on your workload. But you specify the actual URL, instead of that name.

@SebastianSchuetze
Copy link

SebastianSchuetze commented Nov 16, 2021

oh okay. Very good point!
Then it would be a really addition if KEDA / Dapr exceptions would be surfaced above the container as part of a new log type and everything around a container app.

@SebastianSchuetze
Copy link

@tomkerkhove it is f***ing working!!!

image

image

@SebastianSchuetze
Copy link

@guythetechie I don't see where you set the value for azure_devops_pool_id in your snippet.

@tomkerkhove
Copy link
Member

Great to hear that it's working and the pool ID seems to be missing indeed!

@guythetechie
Copy link
Author

@SebastianSchuetze - Glad you got it working. Now I know it's possible. :) It would definitely be useful to have KEDA/DAPR logs.

@tomkerkhove - I pass the pool ID as a parameter, but please let me know if I'm doing it wrong. Just to be clear though - the pool ID is a number (as opposed to a GUID), right? If my template looks good, next stop is making sure the values I'm passing are correct.

Here's the full Bicep file, parameters included:

param container_registry_name string
param container_app_environment_name string
param azure_devops_url string
@secure()
param azure_devops_token string
param azure_devops_pool_name string
param azure_devops_pool_id string

var azure_devops_url_secret_name = 'azure-devops-url'
var azure_devops_pool_name_secret_name = 'azure-devops-pool-name'
var azure_devops_token_secret_name = 'azure-devops-token'
var container_registry_password_secret_name = 'container-registry-password'

resource container_registry 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' existing = {
  name: container_registry_name
}

resource container_app_environment 'Microsoft.Web/kubeEnvironments@2021-02-01' existing = {
  name: container_app_environment_name
}

resource container_app 'Microsoft.Web/containerApps@2021-03-01' = {
  name: 'mycontainerapp'
  location: 'eastus'
  properties: {
    kubeEnvironmentId: container_app_environment.id
    configuration: {
      activeRevisionsMode: 'single'
      secrets: [
        {
          name: azure_devops_url_secret_name
          value: azure_devops_url
        }
        {
          name: azure_devops_pool_name_secret_name
          value: azure_devops_pool_name
        }
        {
          name: azure_devops_token_secret_name
          value: azure_devops_token
        }
        {
          name: container_registry_password_secret_name
          value: container_registry.listCredentials().passwords[0].value
        }
      ]
      registries: [
        {
          server: container_registry.properties.loginServer
          username: container_registry.listCredentials().username
          passwordSecretRef: container_registry_password_secret_name
        }
      ]
    }
    template: {
      containers: [
        {
          image: '${container_registry.properties.loginServer}/repo:123'
          name: 'mycontainerapp'
          env: [
            {
              name: 'AZP_URL'
              secretRef: azure_devops_url_secret_name
            }
            {
              name: 'AZP_TOKEN'
              secretRef: azure_devops_token_secret_name
            }
            {
              name: 'AZP_POOL'
              secretRef: azure_devops_pool_name_secret_name
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 10
        rules: [
          {
            name: 'azure-pipelines'
            custom: {
              type: 'azure-pipelines'
              metadata: {
                poolId: azure_devops_pool_id
                targetPipelinesQueueLength: '1'
              }
              auth: [
                {
                  triggerParameter: 'organizationURL'
                  secretRef: azure_devops_url_secret_name
                }
                {
                  triggerParameter: 'personalAccessToken'
                  secretRef: azure_devops_token_secret_name
                }
              ]
            }
          }
        ]
      }
    }
  }
}

And here's the deployed revision's resource JSON in Azure:

{
    "id": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Web/containerApps/***/revisions/***--azbpvln",
    "name": "***--azbpvln",
    "type": "Microsoft.Web/containerapps/revisions",
    "location": "eastus",
    "properties": {
        "createdTime": "2021-11-15T18:56:19+00:00",
        "fqdn": "",
        "template": {
            "containers": [
                {
                    "image": "***.azurecr.io/***:3873",
                    "name": "***",
                    "env": [
                        {
                            "name": "AZP_URL",
                            "secretRef": "azure-devops-url"
                        },
                        {
                            "name": "AZP_TOKEN",
                            "secretRef": "azure-devops-token"
                        },
                        {
                            "name": "AZP_POOL",
                            "secretRef": "azure-devops-pool-name"
                        }
                    ],
                    "resources": {
                        "cpu": 0.5,
                        "memory": "1Gi"
                    }
                }
            ],
            "scale": {
                "minReplicas": 1,
                "maxReplicas": 10,
                "rules": [
                    {
                        "name": "azure-pipelines",
                        "custom": {
                            "type": "azure-pipelines",
                            "metadata": {
                                "poolId": "13",
                                "targetPipelinesQueueLength": "1"
                            },
                            "auth": [
                                {
                                    "secretRef": "azure-devops-url",
                                    "triggerParameter": "organizationURL"
                                },
                                {
                                    "secretRef": "azure-devops-token",
                                    "triggerParameter": "personalAccessToken"
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "active": true,
        "replicas": 1,
        "trafficWeight": 0,
        "name": "***--azbpvln",
        "healthState": "Healthy",
        "provisioningState": "Provisioned"
    }
}

@guythetechie
Copy link
Author

guythetechie commented Nov 16, 2021

@SebastianSchuetze - If you don't mind, what permissions does your token have? Mine has:
image
and
image

@SebastianSchuetze
Copy link

For testing I used full scope. I will reduce the scoping later to see when something is not working. But in general "manage pool" should be enough. The PAR is only used for registering new agents. The agents themselves get short like ved tokens and authenticate themselves with their own tokens.

@guythetechie
Copy link
Author

Thank you. I'll try full scope.

In addition to Read & manage* pool permissions, I thought the token might need permissions to read the pipelines queue so KEDA can autoscale.

@tomkerkhove
Copy link
Member

@tomkerkhove - I pass the pool ID as a parameter, but please let me know if I'm doing it wrong. Just to be clear though - the pool ID is a number (as opposed to a GUID), right? If my template looks good, next stop is making sure the values I'm passing are correct.

The ID should be the pool ID of your Azure DevOps pool - It should not be used randomly. Probably this is your issue then?

@guythetechie
Copy link
Author

@tomkerkhove - I pass the pool ID as a parameter, but please let me know if I'm doing it wrong. Just to be clear though - the pool ID is a number (as opposed to a GUID), right? If my template looks good, next stop is making sure the values I'm passing are correct.

The ID should be the pool ID of your Azure DevOps pool - It should not be used randomly. Probably this is your issue then?

Sorry, my message was unclear. I am not using a random value as the pool ID. I obtained it using a REST API call to Azure DevOps. Just want to confirm that it is supposed to be a number and not a GUID.

@SebastianSchuetze
Copy link

Pool ID can also be found easily in the url when opening a pool
image

@guythetechie
Copy link
Author

I finally got it working. The issue was my Azure Devops URL: it was https:/dev.azure.com/myorganization/ instead of https://dev.azure.com/myorganization. The scaler didn't like the trailing slash.

I had to create an AKS cluster, deploy KEDA on it with my scaler configuration, and look at the KEDA logs to figure out what was going on. As @SebastianSchuetze mentioned, it would be greatly beneficial to surface these logs in Container Apps.

Thanks @tomkerkhove & @SebastianSchuetze for the assistance!

@tomkerkhove
Copy link
Member

Happy to help and sorry for not being helpful.

Are you open to submitting a PR on KEDA to improve the docs?

https://github.com/kedacore/keda-docs

@SebastianSchuetze
Copy link

Gonna have a look at it. Should I update all docs for all doc versions like 2.3, 2.4, 2.5 or just the latest?

@tomkerkhove
Copy link
Member

tomkerkhove commented Nov 19, 2021

All is preferred but I can backport if it's too much.

Thanks!

@audunsolemdal
Copy link

Really happy I found this issue with regards to

+                            },{
+                                "secretRef": "azp-url",
+                                "triggerParameter": "organizationURL"
                            }]

I did the mistake of writing the SecretRef value as the actual value instead of a secret. I was considering going back to Azure container instances until this actually. Too early to say if this fixes all my issues, but my revisions have been really unstable, with stuck pipeline jobs, failing revisions and even revisions disappearing out of the blue.

@SebastianSchuetze
Copy link

@audunsolemdal regarding your problems you might wanna check my update on the blog post.

Container Apps are not ready for jobs, since you need ScaledJobs from kubernetes in order to do many things properly

https://www.razorspoint.com/2021/11/19/scalable-container-based-azure-pipelines-pools-with-azure-container-apps/

@audunsolemdal
Copy link

Thanks for the head's up.

@MykolaPelyp-SoftServe
Copy link

MykolaPelyp-SoftServe commented May 9, 2022

I am unable to comprehend why so simple operation takes so much efforts, guys, can you look? (secrets reducted)
Code below would deploy it and it would functioning in replicas that consist from 1 container and it would ignore that another pipeline is stuck in queue of 1 pipeline. Target is to keep them in 0 and bring up to 2 when queue is 1

    "resources": [
      {
        "type": "Microsoft.App/containerapps",
        "apiVersion": "2022-01-01-preview",
        "name": "[parameters('name')]",
        "location": "[parameters('location')]",
        "dependsOn": [
          "[concat('Microsoft.App/managedEnvironments/', parameters('environmentName'))]"
        ],
        "kind": "containerapps",
        "properties": {
          "configuration": {
            "secrets": "[parameters('secrets')]",
            "registries": "[parameters('registries')]"
          },
          "template": {
            "containers": [
              {
                "image": "myimg",
                "name": "azp-agents",
                "resources": {
                  "cpu": 2,
                  "memory": "4Gi"
                },
                "env": [
                  {
                    "name": "AZP_URL",
                    "secretRef": "azp-url"
                  },
                  {
                    "name": "AZP_TOKEN",
                    "secretRef": "azp-token"
                  },
                  {
                    "name": "AZP_POOL",
                    "secretRef": "azp-pool"
                  },
                  {
                    "name": "AZP_POOLID",
                    "secretRef": "azp-poolid"
                  }
                ]
              }
            ],
            "scale": {
              "minReplicas": "1",
              "maxReplicas": "5",
              "rules": [
                {
                  "name": "name",
                  "custom": {
                    "type": "azure-pipelines",
                    "metadata": {
                      "organizationURLFromEnv ": "AZP_URL",
                      "poolID": "10",
                      "targetPipelinesQueueLength": "1"
                    },
                    "auth": [
                      {
                        "secretRef": "azp-token",
                        "triggerParameter": "personalAccessToken"
                      },
                      {
                        "secretRef": "azp-url",
                        "triggerParameter": "organizationURL"
                      }
                    ]
                  }
                }
              ]
            }
          },
          "managedEnvironmentId": "[parameters('environmentId')]"

@tomkerkhove
Copy link
Member

I suggest reading the material above given the concept of what you are trying to deploy (a job) is not supported on Azure Container Apps since everything is a deamon/deployment.

It will randomly scale in/out in the middle of your running pipelines which makes no sense, you'll need to wait for job-support in ACA

@MykolaPelyp-SoftServe
Copy link

MykolaPelyp-SoftServe commented May 10, 2022

I wonder whether it is related to API type and version of the Container Apps environment, Im using this 2022 preview
"type": "Microsoft.App/managedEnvironments@2022-01-01-preview"

Found out there were this API As well :
Microsoft.Web/kubeEnvironments@2021-02-01

@MykolaPelyp-SoftServe
Copy link

MykolaPelyp-SoftServe commented Jul 15, 2022

It works, scaling process works, even if you have zero active containers in the ContainerApps it would scale from 0 to desired number of containers.

I was getting pool ID in wrong place, you must get pool ID from organisation settings, not project setting.
Pool ID of target Agent Pool can be found under organization settings of Azure DevOps organisation by navigating to Organisational settings -> Agent pools -> and Get pool ID number from the HTTP Link above, example: _settings/agentpools?poolId=1&view=jobs Pool ID would be 1

template.json:

    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "subscriptionId": {
            "type": "String"
        },
        "name": {
            "type": "String"
        },
        "location": {
            "type": "String"
        },
        "environmentId": {
            "type": "String"
        },
        "containers": {
            "type": "Array"
        },
        "secrets": {
            "type": "Array"
        },
        "registries": {
            "type": "Array"
        }
    },
    "resources": [
        {
            "type": "Microsoft.App/containerapps",
            "apiVersion": "2022-01-01-preview",
            "name": "[parameters('name')]",
            "location": "[parameters('location')]",
            "kind": "containerapps",
            "properties": {
                "configuration": {
                    "secrets": "[parameters('secrets')]",
                    "registries": "[parameters('registries')]"
                },
                "template": {
                    "containers": [
                        {
                            "image": "<reducted>",
                            "name": "azp-agents",
                            "resources": {
                                "cpu": 2,
                                "memory": "4Gi"
                            },
                            "env": [
                                {
                                    "name": "AZP_URL",
                                    "secretRef": "azp-url"
                                },
                                {
                                    "name": "AZP_TOKEN",
                                    "secretRef": "azp-token"
                                },
                                {
                                    "name": "AZP_POOL",
                                    "secretRef": "azp-pool"
                                },
                                {
                                    "name": "AZP_POOLID",
                                    "secretRef": "azp-poolid"
                                }
                            ]
                        }
                    ],
                    "scale": {
                        "minReplicas": "0",
                        "maxReplicas": "2",
                        "rules": [
                            {
                                "name": "<reducted>",
                                "custom": {
                                    "type": "azure-pipelines",
                                    "metadata": {
                                        "organizationURLFromEnv ": "AZP_URL",
                                        "AZP_POOL": "Default",
                                        "poolID": "1",
                                        "targetPipelinesQueueLength": "1"
                                    },
                                    "auth": [
                                        {
                                            "secretRef": "azp-token",
                                            "triggerParameter": "personalAccessToken"
                                        },
                                        {
                                            "secretRef": "azp-url",
                                            "triggerParameter": "organizationURL"
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },
                "managedEnvironmentId": "[parameters('environmentId')]"
            }
        }
    ]
}```

**params json should have secrets with values of your use case, template would reference them.**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants