metrics merging not working in 1.13

[infa-ddeore]

Bug Description

istio sidecar doesn't provide application metrics on :15090/stats/prometheus

istio 1.13.4 is installed on k8 v1.21.11 running on minikube, below sample app is deployed with sidecar injection

istio installation command:

istio-1.13.4/bin/istioctl install --set profile=minimal

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: prometheus-example-app
  name: prometheus-example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-example-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus-example-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: prometheus-example-app
        image: quay.io/brancz/prometheus-example-app:v0.3.0
        ports:
        - name: web
          containerPort: 8080

app shows metrics:

$ curl -s 172.17.0.4:8080/metrics |grep -v ^# | head -5
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.005"} 15
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.01"} 15
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.025"} 15
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.05"} 15
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.1"} 15

sidecar logs show annotation confirming the annotations were read

$ k logs prometheus-example-app-5556bc98cf-rfwgv -c istio-proxy |grep scrap 
2022-05-23T16:50:01.674924Z	info	Prometheus scraping configuration: {true  8080}

sidecar doesnt show app metrics

$ curl -s 172.17.0.4:15090/stats/prometheus |grep -v ^# | awk -F'{' '{print $1}' | sort | uniq | sort |grep -vE 'envoy|istio'
metric_cache_count

no scraping metrics showing that sidecar is trying to scrape application

$ curl -s 172.17.0.4:15090/stats/prometheus |grep -v ^# | grep istio_agent_scrapes

Version

$ ~/tmp/istio-1.13.4/bin/istioctl version
client version: 1.13.4
control plane version: 1.13.4
data plane version: 1.13.4 (1 proxies)

$ k version --short
Client Version: v1.23.5
Server Version: v1.21.11

Additional Information

$ ~/tmp/istio-1.13.4/bin/istioctl bug-report

Target cluster context: minikube

Running with the following config:

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2022-05-23 22:25:18.663203 +0530 IST

Cluster endpoint: https://192.168.64.58:8443
CLI version:
version.BuildInfo{Version:"1.13.4", GitRevision:"202e88863858342eeb0206944e9d277a545eb194", GolangVersion:"go1.17.10", BuildStatus:"Clean", GitTag:"1.13.4"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
{
Component: "pilot",
Info: version.BuildInfo{Version:"1.13.4", GitRevision:"202e88863858342eeb0206944e9d277a545eb194", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.13.4"},
},
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.13.4}

Fetching proxy logs for the following containers:

default/prometheus-example-app/prometheus-example-app-5556bc98cf-rfwgv/istio-proxy
default/prometheus-example-app/prometheus-example-app-5556bc98cf-rfwgv/prometheus-example-app
istio-system/istiod/istiod-7664dfcb67-9w6d7/discovery

Fetching Istio control plane information from cluster.

Running istio analyze on all namespaces and report as below:
Analysis Report:
Info [IST0118] (Service default/prometheus-example-app) Port name (port: 8080, targetPort: 8080) doesn't follow the naming convention of Istio port.
Creating an archive at /Users/ddeore/tmp/bug-report.tar.gz.
Cleaning up temporary files in /var/folders/s6/8n0t02t17jn8qyw9c71k1q280000gp/T/bug-report.
Done.

[howardjohn]

15090 is envoy's metrics, 15020 is the port for merged metrics

[infa-ddeore]

thanks @howardjohn i can see merged metrics on 15020

$ curl -s 172.17.0.4:15020/stats/prometheus |grep -v ^# | awk -F'{' '{print $1}' | sort | uniq | sort |grep -vE 'envoy|istio'
http_request_duration_seconds_bucket
http_request_duration_seconds_count
http_request_duration_seconds_sum
http_requests_total
metric_cache_count
version