Marathon Service Discovery `portMappings` meta label broken for Marathon 1.5.x

[whoward]

What did you do?
Started prometheus with a marathon service discovery configuration that uses the __meta_marathon_port_mapping_label_<labelname> meta tag in a relabel_config to determine whether or not to keep the matching port for scraping and at which path to scrape:

What did you expect to see?
Targets populated within 1 minute of boot

What did you see instead? Under which circumstances?
No targets populated

Environment
CoreOS 1235.12.0
Amazon AWS EC2

• System information:
  Linux 4.13.9-coreos x86_64

• Prometheus version:
  Tested on multiple versions:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

prometheus, version 1.7.1 (branch: master, revision: 3afb3fffa3a29c3de865e1172fb740442e9d0133)
  build user:       root@0aa1b7fc430d
  build date:       20170612-11:44:05
  go version:       go1.8.3

• Prometheus configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'marathon'
    scrape_interval: '5s'
    marathon_sd_configs:
      - servers: ['http://marathon-service:8080']
    relabel_configs:
      # disable all targets which do not have a METRICS_PATH on them
      - source_labels: [__meta_marathon_port_mapping_label_METRICS_PATH]
        action: keep
        regex: (.+)

      # assign the value of METRICS_PATH to the prometheus target's metrics path
      - source_labels: [__meta_marathon_port_mapping_label_METRICS_PATH]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

• Logs:

level=info ts=2017-11-13T17:24:47.263849987Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2017-11-13T17:24:47.263914012Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2017-11-13T17:24:47.263949629Z caller=main.go:217 host_details="(Linux 4.13.9-coreos #1 SMP Thu Oct 26 03:21:00 UTC 2017 x86_64 ip-10-0-15-95.ec2.internal (none))"
level=info ts=2017-11-13T17:24:47.267971677Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-11-13T17:24:47.267938359Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-11-13T17:24:47.270025324Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2017-11-13T17:24:47.877861771Z caller=main.go:326 msg="TSDB started"
level=info ts=2017-11-13T17:24:47.877924302Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2017-11-13T17:24:47.878519327Z caller=main.go:371 msg="Server is ready to receive requests."

[whoward]

I believe I have already debugged the issue - in Marathon 1.5.x the container.docker.portMappings data has been deprecated and normalized to container.portMappings in API responses (changelog)

However it seems that prometheus only looks for the values of port mappings in container.docker.portMappings which only works In Marathon 1.4.x and below.
https://github.com/prometheus/prometheus/blob/master/discovery/marathon/marathon.go#L348-L351

Fixing this should be as simple as also looking for labels inside container.portMappings until support for Marathon < 1.5.0 is dropped (meaning you can drop searching in the original container.docker.portMappings location)

[whoward]

I'll take care of this - going to put up a PR soon.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.