Prometheus unable to scrape : 16686/jaeger/metrics endpoint

[ghost]

Hi,

# smeena at PX-LON-EN-SM.local in ~/Downloads/istio-1.3.2 [11:51:45]
→ helm version
isClient: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.0", GitCommit:"f6025bb9ee7daf9fee0026541c90a6f557a3e0bc", GitTreeState:"clean"}

# smeena at PX-LON-EN-SM.local in ~/Downloads/istio-1.3.2 [11:51:47]
→ helm list
NAME       	REVISION	UPDATED                 	STATUS  	CHART           	NAMESPACE
hoping-crab	1       	Wed Oct 23 11:28:29 2019	DEPLOYED	prometheus-9.2.0	istio-system
istio-init 	1       	Wed Oct 23 10:39:14 2019	DEPLOYED	istio-init-1.3.2	istio-system

After fresh installation of istio with jaeger with helm, I"m getting following errors in prometheus logs.
Prometheus is unable to scrape this istio jaeger endpoint . http://10.1.0.82:16686/jaeger/metrics and it's continuously trying and failing.

level=warn ts=2019-10-23T10:31:30.561201506Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:31:45.561277378Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:32:00.540980878Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:32:15.540034803Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:32:30.517919872Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:32:45.518270282Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:33:00.496879924Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:33:15.496826338Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:33:30.475022673Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:33:45.47591244Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:34:00.45383473Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:34:15.45376043Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:34:30.436910398Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"
level=warn ts=2019-10-23T10:34:45.433037747Z caller=scrape.go:867 component="scrape manager" scrape_pool=kubernetes-pods target=http://10.1.0.82:16686/jaeger/metrics msg="append failed" err="\"INVALID\" is not a valid start token"

I did port-forwarding to try to hit the endpint manually, but this jaeger/metrics isn't a valid endpoint.

`
kubectl port-forward --address 0.0.0.0 -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686

`

https://github.com/istio/istio/blob/1.3.3/install/kubernetes/helm/istio/charts/tracing/templates/deployment-jaeger.yaml#L29-L31

[ghost]

Duplicate of #17264