vm_promscrape proxy_url use wrong Host header

[jtorkkel]

When vm use proxy_url the Host header contains proxy server host address, instead of target service host name.
curl and prometheus set target service address to Host header, which is apparently the right way.

proxy_url added this way

- job_name: docker
  proxy_url: http://ap-tools.xxx.net:3128
  static_configs:
    - targets: ['ap-dock.xxx.net:3333'] 
  #kubernetes_sd_configs

request from vm:

GET http://ap-dock.oneadr.net:3333/metrics HTTP/1.1
User-Agent: vm_promscrape
Host: ap-tools.xxx.net:3128
Accept: text/plain;version=0.0.4;q=1,*/*;q=0.1
X-Prometheus-Scrape-Timeout-Seconds: 30.000
Accept-Encoding: gzip

Request from prometheus and using "curl -x http://ap-tools.xxx.net:3128 http://ap-dock.oneadr.net:3333/metrics:"

GET http://ap-dock.oneadr.net:3333/metrics HTTP/1.1
User-Agent: vm_promscrape
Host: ap-tools.xxx.net:3128
Accept: text/plain;version=0.0.4;q=1,*/*;q=0.1
X-Prometheus-Scrape-Timeout-Seconds: 30.000
Accept-Encoding: gzip

proxy_url allow scraping metrics through proxy service. Needed to scrape metrics outside k8s or docker utilizing proxy installed inside the cluster as example.

Some proxies like NGINX are not handling VM request request correctly by default (or require complex config), prometheus request is easier to handle as host header is already correct. Some proxieslike squid can handle VM request, apparently it regenerate host header automatically.

To Reproduce
Add proxy_url and check outgoing Host header value
Steps to reproduce the behavior.

Expected behavior
target service Host header should be added to request

Logs
Check if any warnings or errors were logged by VictoriaMetrics components
or components in communication with VictoriaMetrics (e.g. Prometheus, Grafana).

Version
The line returned when passing --version command line flag to the binary. For example:

$ ./victoria-metrics-prod --version
victoria-metrics-20220621-070434-tags-v1.78.0-0-g091408be6

Used command-line flags
--promscrape.config=prometheus.yml

[f41gh7]

As workaround, try to use streamParse: true for this target.

job_name: docker
proxy_url: http://ap-tools.xxx.net:3128/
stream_parse: true
static_configs:
    targets: ['ap-dock.xxx.net:3333']
    #kubernetes_sd_configs

[valyala]

The issue must be fixed in the commit f97355d . This commit will be included in the next release. In the meantime it is possible to build vmagent from this commit according to these instructions and verify whether it properly sets Host header when scraping targets via http proxy.

[jtorkkel]

I will validate once back from holiday.

[valyala]

@jtorkkel , if you run vmagent in Docker or Kubernetes, then you can try the following vmagent image with the bugfix - docker.io/victoriametrics/vmagent:heads-master-0-gf9500abfe

[valyala]

FYI, vmagent should properly set Host header when scraping targets via http proxy starting from v1.79.0. Closing the issue as fixed.

@jtorkkel , feel free re-opening this issue or creating new one if you experience the issue in v1.79.0.

[jtorkkel]

verified to work using in 1.79.0 single node version