adding support for generic service discovery

[mad01]

idea

adding support for a generic SD (service discovery) interface like using the same format as file_sd_config but as SD over http.

usecase

you have a existing SD (service discovery) solution not supported and like to write a small service to bridge yours to still work with prometheus.

implementation

/targets GET with a payload of json same as file_sd_config

[
  {
    "targets": [ "node:9104" ],
    "labels": {
      "env": "prod",
      "job": "service"
    }
  }
]

[brian-brazil]

We already have support for generic service discovery via file_sd_config, it doesn't make sense to have two of them. You can use a cronjob to lay down the file for example.

[jamtur01]

This is mostly an aside but I am interested to note that this has come up here and #1675, #2514, #3602, and #3658. I get that you can automate file_sd_discovery with a cronjob but I think there is actually a use case here, especially for folks in service provider, enterprise, multi-tenant situations. Obviously, Prometheus can't be everything to everyone but occasionally I think there's a persistent pain point that appears that might be worth revisiting.

[brian-brazil]

This has been discussed multiple times. We only want one generic way to do things, and have to presume that users have basic infrastructure otherwise we end up expanding being just being a monitoring system.

[krasi-georgiev]

we have discussed this in the group and based on the user's feedback so far http vs file sd doesn't seem to offer much benefits.

if your use case is different you can post it in the group and it will be considered.
https://groups.google.com/forum/#!topic/prometheus-developers/JcE5nSbCe4k

[jamtur01]

@brian-brazil I guess that it has been discussed so many times is my point. :) And I get the issue of scope creep. Perhaps, in light of the thoughts in https://github.com/prometheus/prometheus/tree/master/discovery, the discussion might be moot in a pluggable SD world? But as I said - an aside.

[brian-brazil]

It is pluggable already - via file SD configs. I'm not inclined to add features that could be handled by users by a few lines of shell.

[jamtur01]

@krasi-georgiev I do worry about that though. The previous discussions on this have been shut down pretty fast. I can perhaps see why users with specific use cases might not be so forthcoming. There's a definite "our way or nothing" feel to some of the discussions on Prometheus design. I don't fundamentally have a problem with that approach; a lot of software has been successful because of the adopting that model. Solomon saying no to lots of ideas in early Docker is a good example. :) But it does tend to sometimes stifle discussion or desire to discuss.

[jamtur01]

@brian-brazil Sure. I think Prometheus has a pretty clear idea of who its user is and you're likely correct in assuming folks can solve it that way. I certainly did. :)

[brian-brazil]

But it does tend to stifle discussion or desire to discuss.

It's not really possible to give the same level of explanation the 20th time a discussion comes up versus the 1st. To me this matter was settled years ago. You might have noticed those 5 issues, I've seen all of them.

[krasi-georgiev]

@jamtur01 I also think http based SD would be useful , but the group post suggests that the current file SD is good enough.

you might also look into this PR as an alternative to the cronjob suggestion.
#3720

[jamtur01]

@brian-brazil It was a concern more than a critique. I've definitely been in your shoes - there was a time I could remember every ticket logged about Puppet. That view does sometimes assume nothing changes in the meantime about the tool, environments, ecosystems, etc. Then that can sometimes create a fine line between being open to new ideas and dictating "not invented here". But this is the wrong place for that discussion. Definitely involves buying drinks! Cheers.

[jamtur01]

@krasi-georgiev Thanks - interesting read!

[krasi-georgiev]

@jamtur01 @mad01 can your use case be solved using the cronjob or the alternative PR I posted?
If yes than everyone is happy 😃
if not please post in the group and it is a good start to build up enough requests to reconsider it.

[brian-brazil]

Anyone who can write a service to expose an HTTP api can make it write to disk instead.

[mad01]

@krasi-georgiev of corse it can be solved with file sd and a cron job. it would just have been nice use a http endpoint instead. i wouldn't mind to have spent the time to implement it either.

[krasi-georgiev]

@mad01 did you see #3720
this is much nicer than the cronjob.
It should be just few lines to implement the http endpoint.
if you have time to implement the HTTP adapter can still reconsider http vs file sd again based on how popular the HTTP adapter gets.

[ghadishayban]

Anyone who can write a service to expose an HTTP api can make it write to disk instead.

Very true @brian-brazil . That's how I've done it for a while. I think a lot of other users see the cron to disk as workable but duct-tapey. File_sd is so close to a data API but I think people would like to point the daemon to http://something/inventory and get out of the business of cron and file manip (especially when running in k8s or AWS Fargate and scraping external inventory)

[krasi-georgiev]

@ghadishayban I started the same discussion a while back and it seems that while your use case is valid other uses suggest file_sd as better suited.

Feel free to add your use case to the discussion though.
https://groups.google.com/forum/#!msg/prometheus-developers/JcE5nSbCe4k/Z0ChZaR7AgAJ;context-place=forum/prometheus-developers

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.