Unable to get the queue value returns from external metrics

[toyota790]

Describe the bug
I have followed the example of how to scale using Service Bus Queue as an external metric. But I was unable to get the queue value by running the following command:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queuemessages"
Error from server (BadRequest): metricName is required

The external metric API seems to work well.

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[]}

I appreciate for your kind assistance. Thank you so much!

To Reproduce

1. Create a AKS cluster through Azure Portal

   • Kubernetes version: 1.13.7
   • Enable RBAC: Yes
   • Network configuration: Basic
   • Enable container monitoring: Yes

2. Validate the Metric Server by running kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" command.

3. Install Helm on AKS

4. Create a service bus in Azure

   • Create a namespace
   • Create a queue
   • Create an auth rules for queue:

5. Enable Access to Azure Resources by configuring a service principal

   • Create a service principal scoped to the resource group the resource you monitoring and assign Monitoring Reader to it:

   az ad sp create-for-rbac -n "adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}
   

   • Deploy the environment variables via secret:

   kubectl create secret generic adapter-service-principal -n custom-metrics --from-literal=azure-tenant-id=<tenantid> --from-literal=azure-client-id=<clientid>  --from-literal=azure-client-secret=<secret>
   

6. Since I don't have the go language environment, I used the Service Bus Explorer to create messages as producer.

7. Configure Secret for consumer pod

8. Deploy Consumer

   • Deploy the consumer:

   kubectl apply -f deploy/consumer-deployment.yaml
   

   • Check that the consumer was able to receive messages:

    kubectl logs consumer-547467bf87-bhdnt
    connecting to queue:  externalq
    setting up listener
    received message:  <?xml version="1.0" encoding="utf-8"?>
    <message>Hi mate, how are you?</message>
   

9. Deploy the adapter

   • Create a namespace

   kubectl create namespace custom-metrics
   

   • Create a Service Principle and Secret:

   az ad sp create-for-rbac -n "azure-k8s-metric-adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}
   

   • Use values from service principle created above to create secret

   kubectl create secret generic azure-k8s-metrics-adapter -n custom-metrics --from-literal=azure-tenant-id=<tenantid> --from-literal=azure-client-id=<clientid>  --from-literal=azure-client-secret=<secret>
   

   • Deploy the adapter with the service principle:

   helm install --name sample-release ../../charts/azure-k8s-metrics-adapter --namespace custom-metrics --set azureAuthentication.method=clientSecret --set azureAuthentication.tenantID=<your tenantid> --set azureAuthentication.clientID=<your clientID> --set azureAuthentication.clientSecret=<your clientSecret> --set azureAuthentication.createSecret=true`
   

   • I can hit the external metric endpoint by verifying with the following command

   kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1
   {
     "kind": "APIResourceList",
     "apiVersion": "v1",
     "groupVersion": "external.metrics.k8s.io/v1beta1",
     "resources": []
   }
   

10. Configure Metric Adapter with metrics

    • Replace the resourceGroup and resourceName in externalmetric.yaml
    • Create ExternalMetric resource with following command

    kubectl apply -f deploy/externalmetric.yaml
    

    • List external metrics via following command

    kubectl get aem
    NAME            AGE
    queuemessages   40s
    

11. Deploy the HPA

    • Deploy the HPA:

    kubectl apply -f deploy/hpa.yaml
    

    • Validate that the HPA is configured

    kubectl get hpa consumer-scaler
    NAME              REFERENCE                    TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
    consumer-scaler   Deployment/consumer-scaler   <unknown>/30   1         10        1          3d2h
    

    • You can also check the queue value returns manually by running:

    kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queuemessages"
    Error from server (BadRequest): metricName is required
    

Expected behavior
I can get the queue value by running the following command:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queuemessages"

Kubernetes version (kubectl version):

• [1.13.7] Running on AKS

Logs (kubectl logs <metric adapter pod id>)
kubectl logs sample-release-azure-k8s-metrics-adapter-69556544c9-c7pt6

E0728 12:53:52.320936       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get external
metrics.azure.com)
E0728 12:53:52.320936       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetr
ics.azure.com)
E0728 12:53:53.322735       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get external
metrics.azure.com)
E0728 12:53:53.323570       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetr
ics.azure.com)
E0728 12:53:54.324518       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get external
metrics.azure.com)
E0728 12:53:54.325311       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetr
ics.azure.com)
E0728 12:53:55.326181       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get external
metrics.azure.com)
E0728 12:53:55.327096       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetr
ics.azure.com)
E0728 12:53:56.335038       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get external
metrics.azure.com)
E0728 12:53:56.335947       1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetr
ics.azure.com)

Additional context
kubectl describe hpa consumer-scaler

kubectl de
scribe hpa consumer-scaler
Name:                              consumer-scaler
Namespace:                         default
Labels:                            <none>
Annotations:                       kubectl.kubernetes.io/last-applied-configuration:
                                     {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"consumer-scaler","namespace":"de...
CreationTimestamp:                 Sun, 28 Jul 2019 22:34:45 +0800
Reference:                         Deployment/consumer
Metrics:                           ( current / target )
  "queuemessages" (target value):  <unknown> / 30
Min replicas:                      1
Max replicas:                      10
Deployment pods:                   1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/queuemessages/nil: unable to fetch metrics from external metrics API: metricName is required
Events:
  Type     Reason                        Age                     From                       Message
  ----     ------                        ----                    ----                       -------
  Warning  FailedComputeMetricsReplicas  6m33s (x12 over 9m19s)  horizontal-pod-autoscaler  failed to get external metric queuemessages: unable to get external metric default/queuemessages/nil: unable to fetch metrics from external metrics API: metricName is required
  Warning  FailedGetExternalMetric       4m16s (x21 over 9m19s)  horizontal-pod-autoscaler  unable to get external metric default/queuemessages/nil: unable to fetch metrics from external metrics API: metricName is required

[toyota790]

Based on the logs from the pod of the adapter, it seems like it still queries the ExternalMetric and CustomMetric API with the old version (v1alpha1). I found both of that has already moved to v1alpha2 from v1alpha1 around 5 months ago, but the docker image seems like still leave the old version. Should we need to update the docker image? The last image was created at the end of last year.

[jsturtevant]

Thanks for the detailed report.

You are correct it is the wrong image.
Something went wrong with the release automation. I will look into it. In the mean time to try it out you can build it manually.

[toyota790]

@jsturtevant Got it. Thank you so much!

[ashishchettri]

Hi I am facing the same issue, looking at the logs I can see that it is still using v1alpha1 to query custommetrics.azure.com. Please can you give suggest a workaround for this. I've posted the logs below.
E0828 03:14:37.034658 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetrics.azure.com) E0828 03:14:37.235328 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get externalmetrics.azure.com) E0828 03:14:38.036096 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetrics.azure.com) E0828 03:14:38.237448 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get externalmetrics.azure.com) E0828 03:14:39.038207 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetrics.azure.com) E0828 03:14:39.239036 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ExternalMetric: the server could not find the requested resource (get externalmetrics.azure.com) E0828 03:14:40.040313 1 reflector.go:205] github.com/Azure/azure-k8s-metrics-adapter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CustomMetric: the server could not find the requested resource (get custommetrics.azure.com)

[ashishchettri]

Built the image locally to pick up the latest changes. Working now 👍

[toyota790]

Hi @jsturtevant,
Would you help to update the image for this case?

[gabrielmcf]

Will anyone from Microsoft looking into this issue? Leaving a latest tag in a deploy is always a cause of problems... There is more than a month this was reported and we are facing the same issue here...

[jsturtevant]

Sorry for the delay. The latest image with the support for v1alpha2 schemas:

docker pull mcr.microsoft.com/k8s/metrics/adapter:v0.7.3-296

You can find all the images version at https://mcr.microsoft.com/v2/k8s/metrics/adapter/tags/list and latest is tagged with the highest version.