Scraping in HTTPS return 404

[commarla]

What did you do?
My team developed a service using the node.js library to export a /metrics.
I added it in my configuration.

What did you expect to see?
A correct scrape of this endpoint.
The /metrics page is displayed well (with a 200 return code) when I click on it from the targets page in prometheus.

What did you see instead? Under which circumstances?
A 404 Error server returned HTTP status 404 Not Found in the targets page.

Environment

• System information:

Linux 3.16.0-4-amd64 x86_64

• Prometheus version:

prometheus, version 1.2.1 (branch: master, revision: dd66f2e94b2b662804b9aa1b6a50587b990ba8b7)
  build user:       root@fd9b0daff6bd
  build date:       20161010-15:58:23
  go version:       go1.7.1

• Prometheus configuration file:

  - job_name: 'my-job'
    scheme: https
    scrape_interval: 5s
    scrape_timeout: 3s
    static_configs:
      - targets: ['my-url:443']

• Logs:
  I see nothing revelant in the logfiles.

[fabxc]

That sounds like an issue with your client application. Can you investigate what the difference between your manual requests and the one Prometheus does is?

In general, HTTPS scraping works.

[murphysean]

Ran into the same problem, my app was a go app.

The route was registered as http.Handle("www.example.com/metrics", promhttp.Handler()) Prometheus kept reporting it was getting a 404. When I clicked the link all was fine. I removed the host portion from the handle route and prometheus reported all was fine.

While I wasn't able to monitor the actual call from prometheus, I'm guessing the Host header is missing or not being correctly set.

[commarla]

Hi,

Thanks for your help. I can't do a analysis because it's HTTPS and tcpdump will not be helpful and I don't have access to the destination server. It's on some cloud paas service.

I can ask to the dev if he can change something on the app side but we're using the node.js lib in some other app and it's working.

The paas seems to be the problem here. Maybe I can open a ticket with them.

Thanks.

[murphysean]

Something else I just found:

I think that when I registered the route www.example.com/metrics it treats that as the literal host header needed. When I click the link the browser is smart enough to strip off the :443 that prometheus attaches. When I registered the route as www.example.com:443/metrics prometheus was able to pick it up.

[brian-brazil]

This appears to be a custom client issue rather than one in Prometheus.

[samber]

Hi guys,

We discovered that this issue is due to our hosting provider.

When contacting the exporter, Prometheus put the port into the Host header. Our provider probably has a custom load balancer, that raises error when the Host header has a port.

We will contact them in order to fix this issue. But even if the RFC let you add the port into the Host header, I think it would be an improvement for you to remove it. https://tools.ietf.org/html/rfc2616#page-129

[rossigee]

I'm now having this issue too. I can confirm using curl that my hosting provider isn't liking the port number being appended to the Host header. Unfortunately, it's unlikely that they will 'fix' this at their end, so I really need to fix Prometheus somehow not to send the port in the first place.

[simonpasquier]

@rossigee the Host header is set in the net.http library when building the HTTP request. Fixing it in Prometheus is possible in theory but this would add extra code to deal with systems that don't comply with the standards in the first place.
Is there anything forbidding you from removing the port in the configuration of the targets?

[brian-brazil]

Have you looked at scraping the target directly? It's generally best to have as little infrastructure between Prometheus and your target.

[rossigee]

@simonpasquier - yep, i understand you'd need to add extra code to override net.http behaviour to work around broken proxies which is not ideal - i tried removing the port specifiers from the targets as specified in the prometheus.yml, but net.http seems to add them back again at request time.

@brian-brazil - the target is hosted by a third party and is behind their broken proxy - i can't put my monitoring VM any closer to it, or expose it in any other way unfortunately :(

[brian-brazil]

If we make our code more complicated to deal with this, it's not unlikely that there is other code out there broken in the opposite way. Have you considered using a forward proxy to tweak the request?

[rossigee]

@brian-brazil - Forward proxy considered, but it's an https endpoint, so it seems i'd have to do a bunch of request rewriting for the proxy to know where to forward the request to. And, as you say 'It's generally best to have as little infrastructure between Prometheus and your target.'

I guess it's more an issue with 'net.http' than it is with Prometheus specifically. Why does it insist on sending the port number when it seems other HTTP clients generally do not.

[brian-brazil]

That sounds like something to bring up with Go.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.