prometheus 2.2.1 marathon_sd_config doesn't work properly

[hbceylan]

What did you do?
I have setup for sd_marathon_config in prometheus.yml

What did you see instead? Under which circumstances?

I saw below error:

level=error ts=2018-04-10T14:07:21.991899244Z caller=marathon.go:187 component="discovery manager scrape" discovery=marathon msg="Error while updating services" err="invalid character '<' looking for beginning of value"

Environment
Prometheus version: 2.2.1
Alert Manager: 0.14.0

• Alertmanager version:

0.14.0

• Prometheus version:

2.2.1

• Prometheus configuration file:

- job_name: marathon
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  marathon_sd_configs:
  - servers:
    - http://marathon.url
    timeout: 30s
    refresh_interval: 30s
  relabel_configs:
  - source_labels: [__meta_marathon_app]
    separator: ;
    regex: ^/(.+)
    target_label: app
    replacement: $1
    action: replace
  - source_labels: [app]
    separator: ;
    regex: (service-a|service-b)
    replacement: $1
    action: keep
  - source_labels: [app]
    separator: ;
    regex: (service-a|service-b)
    target_label: __metrics_path__
    replacement: /prometheus
    action: replace```

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[hbceylan]

@brian-brazil I have analyzed and updated the issue. I think this is a bug. Because this configuration works perfectly before upgraded to 1xx to 2xx.

Could you please re-open issue and could you please check this error?

Thanks,

[brian-brazil]

I see no indication here yet of a bug in Prometheus. What evidence do you have that the issue is with Prometheus rather than your setup?

[hbceylan]

I wrote my prometheus config for marathon_sd_config and prometheus error log in the issue. Did you see any syntax error or anything? If you see an error please let me, I'll appricate this.

Also the configuration works before upgraded 1xx to 2.2.1.. I searched error on google but I can't find anything..

So if it's a bug, ok. If not, could you please help me? :)

Thanks,

[brian-brazil]

I've never used Marathon, so I can't help you debug this. I would guess that you're hitting a HTML page where JSON is expected.

[hbceylan]

I have checked before and now. So the marathon job request "http://marathon.url/v2/apps/?embed=apps.tasks" url and this url is JSON.

[adamdecaf]

@hbceylan I could help debug. Do you have any more of an error? Does your marathon require auth?

[hbceylan]

Hi @adamdecaf , I'm getting only err="invalid character '<' looking for beginning of value" error in prometheus logs. Marathon doesn't have any authentication.

BTW, I'm using dockerize prometheus version 2.2.1

[adamdecaf]

@hbceylan What marathon version are you on?

Just to confirm, it sounds like you can load http://marathon.url/v2/apps/?embed=apps.tasks outside of prometheus fine? (json apps array response) In a browser and something like curl?

[hbceylan]

I have tried marathon 0.13.1 and 1.5.6 version.

Yep, http://marathon.url/v2/apps/?embed=apps.tasks url works fine and load metrics as json format.

[mciura]

I guess the OP stumbled upon the issue with basic_auth I reported on prometheus-users on 2 October 2017: https://groups.google.com/d/msg/prometheus-users/TIq6YYb2OXw/dL-q2ahEBgAJ and resolved in my subsequent messages.

The confusion and guesswork stem from a suboptimal error message in Prometheus that only mentions the < character and does not reveal the offending URL. While my PR introducing basic_auth support to marathon_sd_config was rejected, its part that made the URL explicit: https://github.com/prometheus/prometheus/pull/3238/files#diff-a291489ad4c2c7ca260f91adbd181c03R276 still looks like a worthwhile addition to Prometheus.

[adamdecaf]

Were you having problems with token= being added to the header value? I am running from the master branch (changes going into 2.2.x / 2.3.x).

I'm able to use HTTP basic auth with the following config:

marathon_sd_configs:
  - servers:
    - "https://marathon.foo.com"
    - "https://marathon.foo.com"
    basic_auth:
      username: $username
      password_file: /etc/prometheus/secrets/marathon-exporter/password

[mciura]

@adamdecaf Sure, now Prometheus implements basic_auth. When I had the problems, Prometheus was still at version 1.7.2 without support for basic_auth. What I am saying is: please consider putting this snippet at the end of fetchApps() to make the life of the users easier:

apps, err := parseAppJSON(body)
if err != nil {
	return nil, errors.New(err.Error() + " in " + url)
}
return apps, err

[adamdecaf]

@mciura Sure thing, I created #4171 for that.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.