FailedGetExternalMetric event with reason The subscription 'resourceGroups' could not be found

[tomkerkhove]

I was testing the helm chart with a sample scenario based on this sample when I noticed that the HPA was not able to fetch the latest metrics:

ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_externalq,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."

Am I doing something wrong here? I have a feeling that parameters are misinterpreted.

Helm install command

helm install --name azure-metrics-adapter --namespace autoscaling-sandbox --set azureAuthentication.method=clientSecret --set azureAuthentication.tenantID=<removed> --set azureAuthentication.clientID=<removed> --set azureAuthentication.clientSecret=<removed> --set defaultSubscriptionId=<removed> .\azure-k8s-metrics-adapter\

HPA Configuration

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
 name: consumer-scaler
spec:
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: consumer
 minReplicas: 1
 maxReplicas: 10
 metrics:
  - type: External
    external:
      metricName: queuemessages
      metricSelector:
        matchLabels:
          metricName: Messages
          resourceGroup: promitor
          resourceName: promitor-messaging
          resourceProviderNamespace: Microsoft.Servicebus
          resourceType: namespaces
          aggregation: Total
          filter: EntityName_eq_externalq
      targetValue: 30

Service Bus Configuration

Namespace lives in the promitor namespace:

Azure AD Configuration

Application has Monitor Reader permissions which is enough for Azure Monitor.

Kubernetes Logs

❯ kubectl describe hpa --namespace autoscaling-sandbox consumer-scaler
Name:                              consumer-scaler
Namespace:                         autoscaling-sandbox
Labels:                            <none>
Annotations:                       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"consumer-scaler","namespace":"autoscaling-san...
CreationTimestamp:                 Thu, 16 Aug 2018 17:03:18 +0200
Reference:                         Deployment/consumer
Metrics:                           ( current / target )
  "queuemessages" (target value):  <unknown> / 30
Min replicas:                      1
Max replicas:                      10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_externalq,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."
Events:
  Type     Reason                        Age              From                       Message
  ----     ------                        ----             ----                       -------
  Warning  FailedGetExternalMetric       4m (x3 over 5m)  horizontal-pod-autoscaler  unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_orders,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."
  Warning  FailedComputeMetricsReplicas  4m (x3 over 5m)  horizontal-pod-autoscaler  failed to get external metric queuemessages: unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_orders,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."
  Warning  FailedGetExternalMetric       4s (x9 over 4m)  horizontal-pod-autoscaler  unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_externalq,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."
  Warning  FailedComputeMetricsReplicas  4s (x9 over 4m)  horizontal-pod-autoscaler  failed to get external metric queuemessages: unable to get external metric autoscaling-sandbox/queuemessages/&LabelSelector{MatchLabels:map[string]string{aggregation: Total,filter: EntityName_eq_externalq,metricName: Messages,resourceGroup: promitor,resourceName: promitor-messaging,resourceProviderNamespace: Microsoft.Servicebus,resourceType: namespaces,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: insights.MetricsClient#List: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="SubscriptionNotFound" Message="The subscription 'resourceGroups' could not be found."

[jsturtevant]

It looks like the subscirption id is not set. Is the deployment in azure?

Can you run kubectl logs -n custom-metrics <adpater pod name>? If there is no subid is set then you will see an error at the top of the logs. If logging level is over 2 then it should output the subscirption that it was connected to.

One way around this until resolved is to pass the subid as a label on the HPA

@marc-sensenich fyi

[tomkerkhove]

I'm currently running on a local Kubernetes cluster managed by Docker for Windows. I'm passing the default subscription id via helm install, --set defaultSubscriptionId=<removed>.

Looks like variable is set:

[marc-sensenich]

This could be a bad version of the image being run in the development Helm chart. Try with the newest versioned release image. It looks like the old latest is from 17 days ago.

helm install --name azure-metrics-adapter --namespace autoscaling-sandbox --set azureAuthentication.method=clientSecret --set azureAuthentication.tenantID=<removed> --set azureAuthentication.clientID=<removed> --set azureAuthentication.clientSecret=<removed> --set defaultSubscriptionId=<removed> --set image.repository=jsturtevant/azure-k8s-metrics-adapter-amd64 ---set image.tag=v0.3.7-85 .\azure-k8s-metrics-adapter\

[jsturtevant]

Yes sorry. I change the repo name in the build scripts in prep for moving it to an repository (it will eventually not be on jsturtevant but a microsoft repo)

[tomkerkhove]

Good point - Will test this tomorrow!

[tomkerkhove]

Using 0.3.0-alpha.51 works, thanks 👍