custom metrics some time not available, return 404

[pawankkamboj]

custom metrics sometime return 404

deploy 2 node k8s cluster(1.9.1) on bare metal, and setup prometheus adapter for custom metrics, and using k8s metrics server for pods metrics. pods metrics are working fine with HPA and also I am able to autoscale them, but when I am trying to use custom metrics for HPA, sometime not getting result from API server, getting 404. sometime it is working fine.
I am using prometheus JMX exporter to fetch tomcat metrics.
have deploy k8s prometheus adapter here is yaml

#- adapter yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: custom-metrics
namespace: kube-system
labels:
app: custom-metrics
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics
template:
metadata:
labels:
app: custom-metrics
spec:
containers:
- name: custom-metrics
image: registry.text.com/k8s-prometheus-adapter-amd64
args:
- --prometheus-url=http://prometheus.monitoring.svc:9090
- --metrics-relist-interval=30s
- --rate-interval=60s
- --v=4
- --logtostderr=true
ports:
- containerPort: 443
securityContext:
runAsUser: 0

apiVersion: v1
kind: Service
metadata:
name: api
namespace: kube-system
spec:
ports:

• port: 443
  targetPort: 443
  selector:
  app: custom-metrics

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
insecureSkipTLSVerify: true
group: custom.metrics.k8s.io
groupPriorityMinimum: 1000
versionPriority: 5
service:
name: api
namespace: kube-system
version: v1beta1

#- hpa yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: fresco
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: fresco
minReplicas: 1
maxReplicas: 2
metrics:

• type: Pods
  pods:
  metricName: tomcat_threadpool_currentthreadcount
  targetAverageValue: 50

only one pod is running
}[root@docker01 prometheus]# kubectl get pods
NAME READY STATUS RESTARTS AGE
fresco-6c96cc7b57-4977m 1/1 Running 0 43m

but when trying to fetch custom metrics sometime get 404
}[root@docker01 prometheus]# curl -k -u admin:***** https://10.60.1.1/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/jvm_memory_bytes_used"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "fresco-6c96cc7b57-4977m",
"apiVersion": "/__internal"
},
"metricName": "jvm_memory_bytes_used",
"timestamp": "2018-01-29T09:59:45Z",
"value": "434659600"
}
]
}

[root@docker01 prometheus]# curl -k -u admin:****** https://10.60.1.1/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the server could not find the metric jvm_memory_bytes_used for pods",
"reason": "NotFound",
"code": 404
}

#- also it is giving 404 for prometheus adapter api aggregate server
[root@docker01 ~]# curl -k -u admin:AdminPass https://10.60.1.111/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/jvm_memory_bytes_used
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the server could not find the metric jvm_memory_bytes_used for pods",
"reason": "NotFound",
"code": 404

hpa describe
[root@docker01 prometheus]# kubectl describe hpa fresco
Name: fresco
Namespace: default
Labels:
Annotations: autoscaling.alpha.kubernetes.io/conditions=[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-01-29T09:37:00Z","reason":"SucceededGetScale","message":"the HPA controller was able to get...
kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"fresco","namespace":"default"},"spec":{"maxRe...
CreationTimestamp: Mon, 29 Jan 2018 15:06:30 +0530
Reference: Deployment/fresco
Metrics: ( current / target )
"tomcat_threadpool_currentthreadcount" on pods: 30 / 50
Min replicas: 1
Max replicas: 2
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

22m 11m 13 horizontal-pod-autoscaler Warning FailedComputeMetricsReplicas failed to get pods metric value: unable to get metric tomcat_threadpool_currentthreadcount: unable to fetch metrics from API: the server could not find the metric tomcat_threadpool_currentthreadcount for pods
22m 2m 22 horizontal-pod-autoscaler Warning FailedGetPodsMetric unable to get metric tomcat_threadpool_currentthreadcount: unable to fetch metrics from API: the server could not find the metric tomcat_threadpool_currentthreadcount for pods

[pawankkamboj]

scrape_interval was 1m, after setting it to 15s, did not get yet 404