Marathon SD: How to configure when docker network is bridge

[gianluca-sabena]

What did you do?

Deploy an app on DCOS/marathon with network bridge and random ports (I tried all possible combinations between container, host and service set to 0)

What did you expect to see?

Prometheus discovers app

What did you see instead? Under which circumstances?

Nothing, app is not present in target list

Environment

• System information:

  insert output of uname -srm here

• Prometheus version:
  1.7.0

Description

App deployed with network = bridge and a random port has this behaviors:

• Marathon assigns a random port (10180, see example) in json app section (docker.portmappings with labels)
• Marathon assigns a different random port (13759, see example) without labels in embedded task section in marathon json

Since there are no labels in embedded task section marathon sd is not working because I can't use a relabel to filter the correct ports

Example marathon.json used to deploy app:

{
  "id": "grpc-sender", 

  "container": 
  {
    "docker": 
    {
      "image": "....", 
      "forcePullImage": true, 
      "network": "BRIDGE", 
      "portMappings": [
        {
          "containerPort": 0, 
          "hostPort": 0, 
          "servicePort": 0, 
          "protocol": "tcp", 
          "labels": 
          {
            "name": "jmx"
          }, 
          "name": "jmx"
        }
      ]
    }, 
    "type": "DOCKER", 
    "volumes": []
  }, 
  "cpus": 1.0, 
  "mem": 4096, 
  "instances": 1, 


  "cmd": "..."
}

Example marathon api output:

{
id: "grpc-sender",
cmd: "...",
instances: 1,
cpus: 1,
mem: 4096,
disk: 0,
gpus: 0,
container: {
type: "DOCKER",
docker: {
        image: "registry-ci.rnd.unicredit.eu/boilerplate/grpc:rolling",
        network: "BRIDGE",
        portMappings: [
                {
                    containerPort: 0,
                    hostPort: 0,
                    servicePort: 10180,
                    protocol: "tcp",
                    name: "jmx",
                    labels: {
                      name: "jmx"
                    }
                }
              ],
        privileged: false,
        parameters: [],
        forcePullImage: true
    }
  },

ports: [
    10180
    ],
portDefinitions: [
    {
      port: 10180,
      protocol: "tcp",
      labels: {}
    }
  ],
requirePorts: false,

tasks: [
    {
      id: "...",
      slaveId: "...",
      host: "...",
      state: "TASK_RUNNING",
      startedAt: "2017-06-21T07:23:13.370Z",
      stagedAt: "2017-06-21T07:23:10.469Z",
      ports: [
              13759
            ],
      version: "2017-06-21T07:23:10.421Z",
      ipAddresses: [
              {
                ipAddress: "172.17.0.6",
                protocol: "IPv4"
              }
            ],
      appId: "grpc-sender",
      healthCheckResults: [... ]
    }
  ]
}

[gianluca-sabena]

Prometheus adds 2 different label.

I solved my problem joining this 2 labels

  - job_name: 'marathon-apps'
    marathon_sd_configs:
      - servers:
        - http://guinness-m01.rnd.unicredit.eu:8080
        bearer_token: "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1aWQiOiJqZW5raW5zLWV4Y2VlZCIsImV4cCI6MTQ5Nzk3MDM1N30.qRBiBRQK6fS1yj5Jo1FHiPMr_k9tQPHZTEV2R4N-5Ut04mdOI2FxjouWFc9aL9Hg4Y_jxr8UVF_uTNvuagsl4yCLUXw6E7xwCYPjMeKkIiCjJHj427M9PqULH8Gw5Z1GTN1PqRP-6yTcuNV8cEfQ5Uz6ERhUqx1ALsSCv07w6AIhm-uQbL-xc5GyKdD7J7HNccDaPbWxZYv_kxQLiORGVRaeCk_yQ3noHrByoRTOpdNKsSVdNXdm2nMSosvVpQ3QmceNkVNF00GDaf7SS49c_oGzi0PCAosuvYtaHckAivJEhjA6qz5XUSBXcahEFvcFQnP-xHDU99rJ0QP98R9TrA"
    
    relabel_configs:
    - source_labels: [__meta_marathon_app_label_METRICS_PROTOCOL]
      regex:         prometheus
      action:        keep   
    - source_labels: [__meta_marathon_port_definition_label_name, __meta_marathon_port_mapping_label_name]
      separator:     ''   
      regex:         metrics|jmx|metricsmetrics|jmxjmx
      action:        keep                 
    - source_labels: [__meta_marathon_app]
      regex:         \/[^\/]+\/([^\/]+)\/.*
      target_label:  env
      replacement:   ${1}
    - source_labels: [__meta_marathon_app]
      regex:         (.*)
      target_label:  app
      replacement:   ${1}
    - source_labels: [__meta_marathon_task]
      regex:         (.*)
      target_label:  task
      replacement:   ${1}
    - source_labels: [__meta_marathon_image]
      regex:         (.*)
      target_label:  image
      replacement:   ${1}

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.