marathon_sd connect to wrong ports

[tommarute]

Bug Report

What did you do?

Tried to get metrics of marathon applications.

What did you expect to see?

Metrics were scraped by their correct port.

What did you see instead? Under which circumstances?

Got connection refused error because of marathon_sd connect to wrong ports. marathon_sd should use Tasks.Ports instead of PortDefinitions.Ports if RequirePorts is false according to document of marathon.

Environment

• System information:

  Darwin 16.5.0 x86_64

• Prometheus version:

  prometheus, version 2.6.0 (branch: HEAD, revision: dbd1d58c894775c0788470944b818cc724f550fb)
  build user:       root@localhost
  build date:       20181220-09:35:49
  go version:       go1.11.4
  

• Prometheus configuration file:

  Click to expand

  
  global:
  scrape_interval:     15s
  evaluation_interval: 15s
  alerting:
  alertmanagers:
  - static_configs:
    - targets:
  rule_files:
  scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'marathon'
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    marathon_sd_configs:
      - servers:
        - http://localhost:8080
        refresh_interval: 30s
    relabel_configs:
    - source_labels:
    ...
  
  

• Logs:

  level=debug ts=2018-12-20T09:43:09.582113Z caller=scrape.go:825 component="scrape manager" scrape_pool=marathon target=http://192.168.0.1:10006/metrics msg="Scrape failed" err="Get http://192.168.0.1:10006/metrics: dial tcp 192.168.0.1:10012: connect: connection refused"
  

  In this case, correct port was 31690.

  Click to expand

  
  $ curl -s http://localhost:8080/v2/apps/test/httpserver | jq .
  {
    "app": {
      "id": "/test/httpserver",
      ...
      "container": {
        "type": "DOCKER",
        "volumes": [],
        "docker": {
          "image": "centos",
          "network": "HOST",
          "portMappings": [],
          "privileged": false,
          "parameters": [],
          "forcePullImage": false
        }
      },
      ...
      "ports": [
        10006
      ],
      "portDefinitions": [
        {
          "port": 10006,
          "protocol": "tcp",
          "name": "http",
          "labels": {}
        }
      ],
      "requirePorts": false,
      ...
      "tasks": [
        {
          "ipAddresses": [
            {
              "ipAddress": "192.168.0.1",
              "protocol": "IPv4"
            }
          ],
          ...
          "state": "TASK_RUNNING",
          "ports": [
            31690
          ],
          ...
          "host": "192.168.0.1"
        }
      ]
    }
  }
  
  

[tommarute]

FYI: Here is my fix.
tommarute@07323e8

[simonpasquier]

Would you like to send a PR? It should be made againt the release-2.6 branch so we can have a patch release with the fix.

cc @rjanovski @adrien-f @ti-mo

[ti-mo]

Indeed, that reasoning is correct. Code looks good at first glance. Not much time to look into this right now, but feel free to mention me in the PR as well

[tommarute]

Thank you for your reply. I'll send a PR later.

[ti-mo]

@tommarute There is definitely a bug here, but just so I understand your use case better: did you explicitly specify "port": 10006, in the portDefinition, or did you specify 0?

[tommarute]

@ti-mo I specified 0 when I configured marathon.

[simonpasquier]

Closed by #5026