Not able to use nodeport to get metrics from node-exporter endpoints port

[dcvtruong]

Bug Report

What did you do?
Using nodeport to get metrics from node-exporter port 9100 from outside the container
What did you expect to see?
Able to retrieve node-exporter metrics using nodeport port like this "curl http://endpoints_ip:nodeport_port/metrics"
What did you see instead? Under which circumstances?
timeout from the curl command
Environment

• System information:

  Linux 3.10.0-957.1.3.el7.x86_64 x86_64
  K8s 1.13

• Prometheus version:

  Prometheus 2.6.1

• Alertmanager version:

  not relevant to the issue

• Prometheus configuration file:

        prometheus.yml:
              rule_files:
                - /etc/config/rules
                - /etc/config/alerts

              scrape_configs:
                - job_name: prometheus
                  static_configs:
                    - targets:
                      - localhost:9090

                - job_name: 'kubernetes-apiservers'

                  kubernetes_sd_configs:
                    - role: endpoints

                  scheme: https
           
                  tls_config: 
                    insecure_skip_verify: true
                  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

                  relabel_configs:
                    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
                      action: keep
                      regex: default;kubernetes;https

                - job_name: 'kubernetes-nodes'

                  scheme: https

                  tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: true
                  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

                  kubernetes_sd_configs:
                    - role: node

                  relabel_configs:
                    - action: labelmap
                      regex: __meta_kubernetes_node_label_(.+)
                    - target_label: __address__
                      replacement: kubernetes.default.svc:443
                    - source_labels: [__meta_kubernetes_node_name]
                      regex: (.+)
                      target_label: __metrics_path__
                      replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor

                - job_name: 'kubernetes-service-endpoints'

                  kubernetes_sd_configs:
                    - role: endpoints

                  relabel_configs:
                    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
                      action: keep
                      regex: true
                    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
                      action: replace
                      target_label: __scheme__
                      regex: (https?)
                    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
                      action: replace
                      target_label: __metrics_path__
                      regex: (.+)
                    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
                      action: replace
                      target_label: __address__
                      regex: ([^:]+)(?::\d+)?;(\d+)
                      replacement: $1:$2
                    - action: labelmap
                      regex: __meta_kubernetes_service_label_(.+)
                    - source_labels: [__meta_kubernetes_namespace]
                      action: replace
                      target_label: kubernetes_namespace
                    - source_labels: [__meta_kubernetes_service_name]
                      action: replace
                      target_label: kubernetes_name
                    - source_labels: [__meta_kubernetes_pod_node_name]
                      action: replace
                      target_label: kubernetes_node

                - job_name: 'prometheus-pushgateway'
                  honor_labels: true

                  kubernetes_sd_configs:
                    - role: service

                  relabel_configs:
                    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                      action: keep
                      regex: pushgateway

                - job_name: 'kubernetes-services'

                  metrics_path: /probe
                  params:
                    module: [http_2xx]

                  kubernetes_sd_configs:
                    - role: service

                  relabel_configs:
                    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
                      action: keep
                      regex: true
                    - source_labels: [__address__]
                      target_label: __param_target
                    - target_label: __address__
                      replacement: blackbox
                    - source_labels: [__param_target]
                      target_label: instance
                    - action: labelmap
                      regex: __meta_kubernetes_service_label_(.+)
                    - source_labels: [__meta_kubernetes_namespace]
                      target_label: kubernetes_namespace
                    - source_labels: [__meta_kubernetes_service_name]
                      target_label: kubernetes_name

                - job_name: 'kubernetes-pods'

                  kubernetes_sd_configs:
                    - role: pod

                  relabel_configs:
                    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                      action: keep
                      regex: true
                    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                      action: replace
                      target_label: __metrics_path__
                      regex: (.+)
                    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                      action: replace
                      regex: ([^:]+)(?::\d+)?;(\d+)
                      replacement: $1:$2
                      target_label: __address__
                    - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                    - source_labels: [__meta_kubernetes_namespace]
                      action: replace
                      target_label: kubernetes_namespace
                    - source_labels: [__meta_kubernetes_pod_name]
                      action: replace
                      target_label: kubernetes_pod_name

          extraScrapeConfigs:

          networkPolicy:
            enabled: false

• node-exporter svc and endpoints:

prometheus-node-exporter        NodePort    10.233.34.186   <none>        9100:32070/TCP   25h

prometheus-node-exporter        192.168.22.5:9100,192.168.22.6:9100,192.168.22.7:9100 + 3 more...   25h

• Logs:

Not relevant

[simonpasquier]

Thanks for your report. It looks as if this is actually a question about usage and not development.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.

[dcvtruong]

Hi, This is not a question about usage. This is an actual problem not able to use nodeport port to forward to node-exporter exposed port 9100.