New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support / Document Prometheus Remote Storage #352

Open
wleese opened this Issue May 8, 2017 · 41 comments

Comments

Projects
None yet
@wleese
Contributor

wleese commented May 8, 2017

What did you do?

Run the getting started guide, read all the docs.

What did you expect to see?

Some mention of arbitrary configuration of prometheus

Basically I want to set up remote storage for prometheus:
https://prometheus.io/docs/operating/configuration/#<remote_write>

Maybe I'm missing something and this is already supported however.

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz May 9, 2017

Member

You can specify an arbitrary configuration for your Prometheus instances by creating a Prometheus object without setting the serviceMonitorSelector. Then you can manage theSecret named prometheus-<prometheus-object-name> yourself. The Prometheus configuration must be under the key prometheus.yaml.

We plan on supporting remote storage configuration one day, but currently it is not supported, and would therefore have to be done with a fully custom configuration for now.

As we've had a couple of questions around this we should probably start a separate user-guide around fully custom Prometheus configuration.

Member

brancz commented May 9, 2017

You can specify an arbitrary configuration for your Prometheus instances by creating a Prometheus object without setting the serviceMonitorSelector. Then you can manage theSecret named prometheus-<prometheus-object-name> yourself. The Prometheus configuration must be under the key prometheus.yaml.

We plan on supporting remote storage configuration one day, but currently it is not supported, and would therefore have to be done with a fully custom configuration for now.

As we've had a couple of questions around this we should probably start a separate user-guide around fully custom Prometheus configuration.

@wleese

This comment has been minimized.

Show comment
Hide comment
@wleese

wleese May 9, 2017

Contributor

Great stuff. I'll close this then as this helps me well enough.

Contributor

wleese commented May 9, 2017

Great stuff. I'll close this then as this helps me well enough.

@wleese wleese closed this May 9, 2017

@fabxc

This comment has been minimized.

Show comment
Hide comment
@fabxc

fabxc May 9, 2017

Contributor

Reopening as we want to allow remote storage at some point. Probably a bit out there as the upstream API/configuration isn't stable yet though.

We might want to make custom config an explicit parameter at some point @brancz
The omitting of serviceMonitorSelector is pretty leaky by now as it also implies skipping of AM config etc.

Contributor

fabxc commented May 9, 2017

Reopening as we want to allow remote storage at some point. Probably a bit out there as the upstream API/configuration isn't stable yet though.

We might want to make custom config an explicit parameter at some point @brancz
The omitting of serviceMonitorSelector is pretty leaky by now as it also implies skipping of AM config etc.

@fabxc fabxc reopened this May 9, 2017

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz May 9, 2017

Member

Agreed.

Member

brancz commented May 9, 2017

Agreed.

@rwkarg

This comment has been minimized.

Show comment
Hide comment
@rwkarg

rwkarg May 9, 2017

Looking to do the same thing, as well. Would appreciate a user-guide around adding to the prometheus.yaml config, specifically adding a remote storage endpoint.

rwkarg commented May 9, 2017

Looking to do the same thing, as well. Would appreciate a user-guide around adding to the prometheus.yaml config, specifically adding a remote storage endpoint.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 17, 2017

Started looking into this as well. It appears that enabling remote_read and remote_write natively via deployment configuration is the simplest path forward. Since with a custom configuration, by disabling serviceMonitorSelector we might have to add the serviceMonitor config by hand into the prometheus configuration.

Even though this is an experimental, feature, it would be nice to have this available in the operator as well.

gdmello commented Aug 17, 2017

Started looking into this as well. It appears that enabling remote_read and remote_write natively via deployment configuration is the simplest path forward. Since with a custom configuration, by disabling serviceMonitorSelector we might have to add the serviceMonitor config by hand into the prometheus configuration.

Even though this is an experimental, feature, it would be nice to have this available in the operator as well.

@wleese

This comment has been minimized.

Show comment
Hide comment
@wleese

wleese Aug 18, 2017

Contributor

natively via deployment configuration

@gdmello would you mind explaining what you mean by this? I'm looking for a nice way to doing this.

Previously I simply statically defined it in the prometheus-operator container - not a solution I'd like to continue using.

Contributor

wleese commented Aug 18, 2017

natively via deployment configuration

@gdmello would you mind explaining what you mean by this? I'm looking for a nice way to doing this.

Previously I simply statically defined it in the prometheus-operator container - not a solution I'd like to continue using.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 19, 2017

@wleese - I meant it would be easier if remote read/write just worked by modifying the prometheus configuration in /contrib/kube-prometheus/manifests/prometheus/prometheus-k8s.yaml (to add remote-read/ remote-write configuration) and deploying the operator with /contrib/kube-prometheus/hack/cluster-monitoring/deploy. Yes, you would have to checkout this repo first.

I forked this repo, and added this feature in, and tested on minikube only (will test in a k8s env next week). You can try it out - gavind/prometheus-operator-dev:787 <- use this image in /prometheus-operator/contrib/kube-prometheus/manifests/prometheus-operator/prometheus-operator.yaml and deploy using the above command.

My source is here.

gdmello commented Aug 19, 2017

@wleese - I meant it would be easier if remote read/write just worked by modifying the prometheus configuration in /contrib/kube-prometheus/manifests/prometheus/prometheus-k8s.yaml (to add remote-read/ remote-write configuration) and deploying the operator with /contrib/kube-prometheus/hack/cluster-monitoring/deploy. Yes, you would have to checkout this repo first.

I forked this repo, and added this feature in, and tested on minikube only (will test in a k8s env next week). You can try it out - gavind/prometheus-operator-dev:787 <- use this image in /prometheus-operator/contrib/kube-prometheus/manifests/prometheus-operator/prometheus-operator.yaml and deploy using the above command.

My source is here.

@wleese

This comment has been minimized.

Show comment
Hide comment
@wleese

wleese Aug 19, 2017

Contributor

@gdmello great stuff, many thanks.

Contributor

wleese commented Aug 19, 2017

@gdmello great stuff, many thanks.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 21, 2017

@brancz - Is this a good start for a PR even though the feature is experimental? LTS is a production concern for organizations adopting prometheus-operator.

gdmello commented Aug 21, 2017

@brancz - Is this a good start for a PR even though the feature is experimental? LTS is a production concern for organizations adopting prometheus-operator.

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Aug 22, 2017

Member

I'm ok with adding it (I haven't had a look at the code, yet but rather mean the general idea), however, we need to explicitly declare in the documentation that everything in this space is highly experimental just like it is in upstream Prometheus, and it can change from any release to the next, we can only support it on a best effort basis.

Out of curiosity, what is your long term storage? I am one a contributor to upstream Prometheus and follow the work around long term storage closely, but haven't observed any solutions that are actually production ready, hence my initial concerns.

Member

brancz commented Aug 22, 2017

I'm ok with adding it (I haven't had a look at the code, yet but rather mean the general idea), however, we need to explicitly declare in the documentation that everything in this space is highly experimental just like it is in upstream Prometheus, and it can change from any release to the next, we can only support it on a best effort basis.

Out of curiosity, what is your long term storage? I am one a contributor to upstream Prometheus and follow the work around long term storage closely, but haven't observed any solutions that are actually production ready, hence my initial concerns.

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Aug 22, 2017

Member

Note though that there are some large changes coming up as we're deprecating the v1alpha1 objects and moving them to v1 (#555).

Member

brancz commented Aug 22, 2017

Note though that there are some large changes coming up as we're deprecating the v1alpha1 objects and moving them to v1 (#555).

@wleese

This comment has been minimized.

Show comment
Hide comment
@wleese

wleese Aug 22, 2017

Contributor

Out of curiosity, what is your long term storage?

I'd gladly add my answer, even though this wasn't addressed to me ;)
Prometheus -> Remote storage adapter -> InfluxCloud

but haven't observed any solutions that are actually production ready

Would you care to elaborate?

Contributor

wleese commented Aug 22, 2017

Out of curiosity, what is your long term storage?

I'd gladly add my answer, even though this wasn't addressed to me ;)
Prometheus -> Remote storage adapter -> InfluxCloud

but haven't observed any solutions that are actually production ready

Would you care to elaborate?

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Aug 22, 2017

Member

Beyond the remote read/write implementation itself being highly experimental, the influx adapter is actually just an example implementation, or at least started as one. Influx is working on integrating real support into influx itself for remote storage purposes, this will be a real solution although the remote read/write interface still remains highly experimental. And this support is still months away to be available in the latest releases of influx.

Again out of curiosity, why do you require long term storage?

Member

brancz commented Aug 22, 2017

Beyond the remote read/write implementation itself being highly experimental, the influx adapter is actually just an example implementation, or at least started as one. Influx is working on integrating real support into influx itself for remote storage purposes, this will be a real solution although the remote read/write interface still remains highly experimental. And this support is still months away to be available in the latest releases of influx.

Again out of curiosity, why do you require long term storage?

@wleese

This comment has been minimized.

Show comment
Hide comment
@wleese

wleese Aug 22, 2017

Contributor

@brancz

why do you require long term storage?

Because we value metrics for various purposes (14mil unique series, 40k/s, 90 day retention by default and more than 1000 grafana dashboards - not autogenerated btw). We're currently building up infra in the cloud and Graphite & Nagios simply doesn't fit our desire to have a low maintenance, cloud friendly (monitoring &) metrics platform (opentsdb didn't have the community, kairodb was too niche, etc, etc).

As for the long retention specifically: we have hundreds of services, some updated daily and some a few times a year. Having metrics that reach far back helps teams that might have inherited an old service to understand what they're working on. Further more, our traffic is very dependent on season. Some of our metrics have a >365d retention time to compare to last year.

Contributor

wleese commented Aug 22, 2017

@brancz

why do you require long term storage?

Because we value metrics for various purposes (14mil unique series, 40k/s, 90 day retention by default and more than 1000 grafana dashboards - not autogenerated btw). We're currently building up infra in the cloud and Graphite & Nagios simply doesn't fit our desire to have a low maintenance, cloud friendly (monitoring &) metrics platform (opentsdb didn't have the community, kairodb was too niche, etc, etc).

As for the long retention specifically: we have hundreds of services, some updated daily and some a few times a year. Having metrics that reach far back helps teams that might have inherited an old service to understand what they're working on. Further more, our traffic is very dependent on season. Some of our metrics have a >365d retention time to compare to last year.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 22, 2017

@brancz - I'm considering Prometheus -> Adapter -> InfluxDB as well.

Again out of curiosity, why do you require long term storage?
Three reasons-

  • Prometheus design philosophy suggests not storing data in the long term. (Based on reading some PRs, I've understood that to mean no more than a month or so.)
  • We are looking at Prometheus to replace possibly one of 2 commercial monitoring/ alerting solutions, and reduce/consolidate our spending.
  • We store business metrics as well (req/s etc), though this might be frowned upon, and so need to look at historical data over the last year or year over year comparisons.

the influx adapter is actually just an example implementation

Why not make the remote/ read write interface and examples mainstream? It would not depart from the design philosophy of staying out of distributed storage.

Influx is working on integrating real support into influx itself for remote storage purposes, this will be a
real solution although the remote read/write interface still remains highly experimental.

This is great to hear! Are there any reservations with using InfluxDB at this point (apart from the read/write interface being experimental)?

gdmello commented Aug 22, 2017

@brancz - I'm considering Prometheus -> Adapter -> InfluxDB as well.

Again out of curiosity, why do you require long term storage?
Three reasons-

  • Prometheus design philosophy suggests not storing data in the long term. (Based on reading some PRs, I've understood that to mean no more than a month or so.)
  • We are looking at Prometheus to replace possibly one of 2 commercial monitoring/ alerting solutions, and reduce/consolidate our spending.
  • We store business metrics as well (req/s etc), though this might be frowned upon, and so need to look at historical data over the last year or year over year comparisons.

the influx adapter is actually just an example implementation

Why not make the remote/ read write interface and examples mainstream? It would not depart from the design philosophy of staying out of distributed storage.

Influx is working on integrating real support into influx itself for remote storage purposes, this will be a
real solution although the remote read/write interface still remains highly experimental.

This is great to hear! Are there any reservations with using InfluxDB at this point (apart from the read/write interface being experimental)?

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Aug 22, 2017

Member

All of the above sounds great! Thanks for sharing!

This is great to hear! Are they any reservations with using InfluxDB at this point (apart from the read/write interface being experimental)?

This is really up to you, whether you are willing to buy into the ecosystem and pay for clustering, as if you have single nodes, then Prometheus on it's own will outperform influx simply because it can optimize the time series database for the concrete use case.

I would strongly suggest to look whether your requirements could be satisfied using Prometheus federation. I know that multiple companies simply federate out the metrics that are important for longer retention times, and this works very well and is much easier than maintaining a centralized distributed time-series database (like influx).

We store business metrics as well (req/s etc), though this might be frowned upon, and so need to look at historical data over the last year or year over year comparisons.

This is a totally legit use case and can very easily be achieved with Prometheus federation, by federating only those metrics that you care about for a year to year comparison. The "what if" argument usually comes into play at this point and I invite you to let go of collectors urge 🙂 .

Prometheus design philosophy suggests not storing data in the long term. (Based on reading some PRs, I've understood that to mean no more than a month or so.)

The way I see it is that the primary use case is that the ingesting Prometheus servers are viewed as a high performance real time monitoring, dashboarding and alerting solution with relatively short retention (whatever retention shows you whether new deployments are successful for example). The most value people get out of monitoring is alerting and detecting patterns longer than 1 month are extremely rare from my experience.

Influx (or any distributed tsdb) seems like an easy solution to all the problems but comes at the cost of having to run a very complicated distributed system (plus likely cost of the actual software). Prometheus federation can help in almost any scenario and makes you realize and think about the metrics you actually care about. Nonetheless we see the wish which is why remote read/write is a thing 🙂 .

As I said, this is ultimately your decision and there are other also totally legit use cases for remote read/write so we're not against introducing it, just want to make sure people explore not as complex alternatives.

Member

brancz commented Aug 22, 2017

All of the above sounds great! Thanks for sharing!

This is great to hear! Are they any reservations with using InfluxDB at this point (apart from the read/write interface being experimental)?

This is really up to you, whether you are willing to buy into the ecosystem and pay for clustering, as if you have single nodes, then Prometheus on it's own will outperform influx simply because it can optimize the time series database for the concrete use case.

I would strongly suggest to look whether your requirements could be satisfied using Prometheus federation. I know that multiple companies simply federate out the metrics that are important for longer retention times, and this works very well and is much easier than maintaining a centralized distributed time-series database (like influx).

We store business metrics as well (req/s etc), though this might be frowned upon, and so need to look at historical data over the last year or year over year comparisons.

This is a totally legit use case and can very easily be achieved with Prometheus federation, by federating only those metrics that you care about for a year to year comparison. The "what if" argument usually comes into play at this point and I invite you to let go of collectors urge 🙂 .

Prometheus design philosophy suggests not storing data in the long term. (Based on reading some PRs, I've understood that to mean no more than a month or so.)

The way I see it is that the primary use case is that the ingesting Prometheus servers are viewed as a high performance real time monitoring, dashboarding and alerting solution with relatively short retention (whatever retention shows you whether new deployments are successful for example). The most value people get out of monitoring is alerting and detecting patterns longer than 1 month are extremely rare from my experience.

Influx (or any distributed tsdb) seems like an easy solution to all the problems but comes at the cost of having to run a very complicated distributed system (plus likely cost of the actual software). Prometheus federation can help in almost any scenario and makes you realize and think about the metrics you actually care about. Nonetheless we see the wish which is why remote read/write is a thing 🙂 .

As I said, this is ultimately your decision and there are other also totally legit use cases for remote read/write so we're not against introducing it, just want to make sure people explore not as complex alternatives.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 22, 2017

Thanks @brancz! will look into Prometheus Federation.

gdmello commented Aug 22, 2017

Thanks @brancz! will look into Prometheus Federation.

@gdmello

This comment has been minimized.

Show comment
Hide comment
@gdmello

gdmello Aug 23, 2017

@brancz - so had a look at federation, and I have 2 concerns with it-

  • If using Grafana, at the very least, I will require 2 dashboards, one with data sourced from the slave node for instance/ granular metrics, and the other, from data sourced from the master node, for charts over a large timespan.
  • Requires each time series to be aggregated and thus configured in the master node, and for a large number of time series this can get tedious. (Of course you are right - it forces you to care about the metrics you care about, but if I care about all my business metrics then this argument doesn't hold water unfortunately)

For LTS, the only viable option seems InfluxDB atm, waiting on their pricing for the Enterprise version.

gdmello commented Aug 23, 2017

@brancz - so had a look at federation, and I have 2 concerns with it-

  • If using Grafana, at the very least, I will require 2 dashboards, one with data sourced from the slave node for instance/ granular metrics, and the other, from data sourced from the master node, for charts over a large timespan.
  • Requires each time series to be aggregated and thus configured in the master node, and for a large number of time series this can get tedious. (Of course you are right - it forces you to care about the metrics you care about, but if I care about all my business metrics then this argument doesn't hold water unfortunately)

For LTS, the only viable option seems InfluxDB atm, waiting on their pricing for the Enterprise version.

@midnightconman

This comment has been minimized.

Show comment
Hide comment
@midnightconman

midnightconman Nov 30, 2017

Contributor

Is it possible to add an additionalConfiguration field? It seems like this type of request and need comes up often. Then the prometheus operator would be able to just append this block to the end of the prometheus configuration.

ex:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  labels:
    prometheus: k8s
spec:
  replicas: 2
  version: v1.7.0
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
  additionalConfiguration: |-
    remote_write:
     - url: "http://localhost:9201/write"
Contributor

midnightconman commented Nov 30, 2017

Is it possible to add an additionalConfiguration field? It seems like this type of request and need comes up often. Then the prometheus operator would be able to just append this block to the end of the prometheus configuration.

ex:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  labels:
    prometheus: k8s
spec:
  replicas: 2
  version: v1.7.0
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
  additionalConfiguration: |-
    remote_write:
     - url: "http://localhost:9201/write"
@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Nov 30, 2017

Member

I’ve mentioned this a couple times, if we allow passing arbitrary arguments then there are no guarantees for upgrades which is a large reason for why the Prometheus operator exists. We will properly integrate remote read has one of the next feature though. It might be experimental in the beginning.

Member

brancz commented Nov 30, 2017

I’ve mentioned this a couple times, if we allow passing arbitrary arguments then there are no guarantees for upgrades which is a large reason for why the Prometheus operator exists. We will properly integrate remote read has one of the next feature though. It might be experimental in the beginning.

@midnightconman

This comment has been minimized.

Show comment
Hide comment
@midnightconman

midnightconman Nov 30, 2017

Contributor

Ok cool, I haven't seen the upgrade concern in the issues I have read and I see that as a very valid concern. Thank you, I look forward to the newer version that supports remote read / write.

Contributor

midnightconman commented Nov 30, 2017

Ok cool, I haven't seen the upgrade concern in the issues I have read and I see that as a very valid concern. Thank you, I look forward to the newer version that supports remote read / write.

@SEJeff

This comment has been minimized.

Show comment
Hide comment
@SEJeff

SEJeff Nov 30, 2017

Yeah for some users (certainly myself included), this is a really important feature. We have a pretty sizable influx setup (load tested to handle peak traffic of 500,000 metrics per second) and want to use influx for long term storage of metrics from prometheus. This lets us do longer term analysis on app metrics we pull via prometheus.

SEJeff commented Nov 30, 2017

Yeah for some users (certainly myself included), this is a really important feature. We have a pretty sizable influx setup (load tested to handle peak traffic of 500,000 metrics per second) and want to use influx for long term storage of metrics from prometheus. This lets us do longer term analysis on app metrics we pull via prometheus.

@cofyc

This comment has been minimized.

Show comment
Hide comment
@cofyc

cofyc Dec 18, 2017

Contributor

hi, guys

I am currently using Prometheus 1.8.2 managed by prometheus-operator, and want to upgrade to prometheus 2.0. Because theirs data formats are not compatible, I followed the migration guide to deploy a new prometheus 2.0 instance and configure it to read old prometheus data, but found it's impossible to configure remote_read in PrometheusSpec.

How about adding a Prometheus2 resource type or version v2 of Prometheus?

Contributor

cofyc commented Dec 18, 2017

hi, guys

I am currently using Prometheus 1.8.2 managed by prometheus-operator, and want to upgrade to prometheus 2.0. Because theirs data formats are not compatible, I followed the migration guide to deploy a new prometheus 2.0 instance and configure it to read old prometheus data, but found it's impossible to configure remote_read in PrometheusSpec.

How about adding a Prometheus2 resource type or version v2 of Prometheus?

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Dec 19, 2017

Member

Certain versions of 1.x also support remote read, we will support remote read/write support with the Prometheus Operator, we just haven't gotten around to it. If anyone wants to take a stab at it, PRs are highly appreciated! (if you need some guidance I'm happy to help, even if anyone is new to Go 🙂)

Member

brancz commented Dec 19, 2017

Certain versions of 1.x also support remote read, we will support remote read/write support with the Prometheus Operator, we just haven't gotten around to it. If anyone wants to take a stab at it, PRs are highly appreciated! (if you need some guidance I'm happy to help, even if anyone is new to Go 🙂)

@trevrosen

This comment has been minimized.

Show comment
Hide comment
@trevrosen

trevrosen Jan 12, 2018

Per these release notes, it would seem that remote_storage is now supported in the latest release. Is this ticket still open pending documentation of feature usage?

trevrosen commented Jan 12, 2018

Per these release notes, it would seem that remote_storage is now supported in the latest release. Is this ticket still open pending documentation of feature usage?

@SEJeff

This comment has been minimized.

Show comment
Hide comment
@SEJeff

SEJeff Jan 12, 2018

RemoteReadSpec RemoteWriteSpec and an example.

Here is an example of how to configure influxdb for this using the newer influx 1.4.x

SEJeff commented Jan 12, 2018

RemoteReadSpec RemoteWriteSpec and an example.

Here is an example of how to configure influxdb for this using the newer influx 1.4.x

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Jan 13, 2018

Member

Yes the feature now exists, but it’s true that we should add some real docs. I’ll leave this open until we have those.

Member

brancz commented Jan 13, 2018

Yes the feature now exists, but it’s true that we should add some real docs. I’ll leave this open until we have those.

@danisabad

This comment has been minimized.

Show comment
Hide comment
@danisabad

danisabad Feb 18, 2018

Any news about proper documentation on this? Or could you please, provide an example how it supposed to be used?

danisabad commented Feb 18, 2018

Any news about proper documentation on this? Or could you please, provide an example how it supposed to be used?

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Feb 26, 2018

Member

I don't have time to write the docs right now, but the generated API docs are up to date: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md

See the remote read/write specs of the PrometheusSpec. If someone figures it out, please contribute docs! 🙂 (as a hint, the config itself is pretty much exactly the same as the upstream ones)

Member

brancz commented Feb 26, 2018

I don't have time to write the docs right now, but the generated API docs are up to date: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md

See the remote read/write specs of the PrometheusSpec. If someone figures it out, please contribute docs! 🙂 (as a hint, the config itself is pretty much exactly the same as the upstream ones)

@yosefy

This comment has been minimized.

Show comment
Hide comment
@yosefy

yosefy Jun 25, 2018

after trying to understand for few hours i still can't simply find a file where i should add the remote configuration lines (like in regular prometheus.yml). can you just point to that file pls? thanks

yosefy commented Jun 25, 2018

after trying to understand for few hours i still can't simply find a file where i should add the remote configuration lines (like in regular prometheus.yml). can you just point to that file pls? thanks

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Jun 25, 2018

Member

Say you have a Prometheus object, like this:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    prometheus: prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager
      port: web

Then the remote storage field just goes under the spec field:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    prometheus: prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager
      port: web
  remoteWrite:
    url: "https://my-remote-write-backend/"
    ... more fields ...
  remoteRead:
    url: "https://my-remote-read-backend/"
    ... more fields ...

As pointed out, the PrometheusSpec contains the remoteWrite and remoteRead fields which have a configuration of their own respectively.

Let us know if that is useful and as always it would be awesome to have anyone new to the matter submit a pull request to improve the docs, as we are very close to the development of these topics so it's often hard to judge what's clear enough and what needs to be elaborated further. Thanks! 🙂

Member

brancz commented Jun 25, 2018

Say you have a Prometheus object, like this:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    prometheus: prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager
      port: web

Then the remote storage field just goes under the spec field:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    prometheus: prometheus
spec:
  replicas: 2
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager
      port: web
  remoteWrite:
    url: "https://my-remote-write-backend/"
    ... more fields ...
  remoteRead:
    url: "https://my-remote-read-backend/"
    ... more fields ...

As pointed out, the PrometheusSpec contains the remoteWrite and remoteRead fields which have a configuration of their own respectively.

Let us know if that is useful and as always it would be awesome to have anyone new to the matter submit a pull request to improve the docs, as we are very close to the development of these topics so it's often hard to judge what's clear enough and what needs to be elaborated further. Thanks! 🙂

@StianOvrevage

This comment has been minimized.

Show comment
Hide comment
@StianOvrevage

StianOvrevage Jun 27, 2018

We want to use the Helm Chart at https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus to deploy Prometheus. How exacly would we go about adding remoteWrite there? I cannot see in in the values.yaml or templates.

StianOvrevage commented Jun 27, 2018

We want to use the Helm Chart at https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus to deploy Prometheus. How exacly would we go about adding remoteWrite there? I cannot see in in the values.yaml or templates.

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Jun 27, 2018

Member

@StianOvrevage the helm chart allows you so specify remoteRead and remoteWrite configurations:

## The remote_read spec configuration for Prometheus.
## Ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec
remoteRead: {}
# remoteRead:
# - url: http://remote1/read
## The remote_write spec configuriation for Prometheus.
## Ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec
remoteWrite: {}
# remoteWrite:
# - url: http://remote1/push

Member

brancz commented Jun 27, 2018

@StianOvrevage the helm chart allows you so specify remoteRead and remoteWrite configurations:

## The remote_read spec configuration for Prometheus.
## Ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec
remoteRead: {}
# remoteRead:
# - url: http://remote1/read
## The remote_write spec configuriation for Prometheus.
## Ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec
remoteWrite: {}
# remoteWrite:
# - url: http://remote1/push

@StianOvrevage

This comment has been minimized.

Show comment
Hide comment
@StianOvrevage

StianOvrevage Jun 28, 2018

Yes, I saw that.

But prometheus-operator/helm/prometheus contains ONLY prometheus right? I'm using prometheus-operator/helm/kube-prometheus which is the umbrella chart for all the components, alertmanager, grafana and prometheus.

What I can't figure out is how to set remoteWrite when using the kube-prometheus umbrella chart. If I don't have to I would like to not manually install and keep up with all the charts referenced in kube-prometheus just to change this one setting.

StianOvrevage commented Jun 28, 2018

Yes, I saw that.

But prometheus-operator/helm/prometheus contains ONLY prometheus right? I'm using prometheus-operator/helm/kube-prometheus which is the umbrella chart for all the components, alertmanager, grafana and prometheus.

What I can't figure out is how to set remoteWrite when using the kube-prometheus umbrella chart. If I don't have to I would like to not manually install and keep up with all the charts referenced in kube-prometheus just to change this one setting.

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz
Member

brancz commented Jun 29, 2018

@StianOvrevage

This comment has been minimized.

Show comment
Hide comment
@StianOvrevage

StianOvrevage Jul 6, 2018

I added a deployPrometheus toggle to the helm chart. In the same process I tried adding/copying over the remoteWrite and remoteRead sections from helm/prometheus/values.yaml to helm/kube-prometheus/values.yamland placed them under the prometheus object.

However when testing they did not show up in the Prometheus resource in K8s.

How do variables get propagated from kube-prometheus/values.yaml to prometheus/ ?

StianOvrevage commented Jul 6, 2018

I added a deployPrometheus toggle to the helm chart. In the same process I tried adding/copying over the remoteWrite and remoteRead sections from helm/prometheus/values.yaml to helm/kube-prometheus/values.yamland placed them under the prometheus object.

However when testing they did not show up in the Prometheus resource in K8s.

How do variables get propagated from kube-prometheus/values.yaml to prometheus/ ?

@tomwilkie

This comment has been minimized.

Show comment
Hide comment
@tomwilkie

tomwilkie Jul 24, 2018

I had a go at this and the following worked, for authenticated remote write:

# kubectl create secret generic --namespace=monitoring prometheus-secret --from-literal=username=username --from-literal=password=password`
# cat <<EOF >values.yaml
prometheus:
  remoteWrite:
    - url: http://foo/bar
      basicAuth:
        username:
          name: prometheus-secret
          key: username
        password:
          name: prometheus-secret
          key: password
EOF
# helm install coreos/kube-prometheus --values config.yaml  --name kube-prometheus --namespace monitoring

HTH

tomwilkie commented Jul 24, 2018

I had a go at this and the following worked, for authenticated remote write:

# kubectl create secret generic --namespace=monitoring prometheus-secret --from-literal=username=username --from-literal=password=password`
# cat <<EOF >values.yaml
prometheus:
  remoteWrite:
    - url: http://foo/bar
      basicAuth:
        username:
          name: prometheus-secret
          key: username
        password:
          name: prometheus-secret
          key: password
EOF
# helm install coreos/kube-prometheus --values config.yaml  --name kube-prometheus --namespace monitoring

HTH

@incognick

This comment has been minimized.

Show comment
Hide comment
@incognick

incognick Aug 12, 2018

I ended up here after searching around for a bit. I'm trying to add the bearer_token to every remote_write request however it is not being added to the request. The url field is working as expected.

When I exec into the prometheus container and cat out the generated config file, only the url is there with a remote_timeout setting.

Anyone run into this yet?

incognick commented Aug 12, 2018

I ended up here after searching around for a bit. I'm trying to add the bearer_token to every remote_write request however it is not being added to the request. The url field is working as expected.

When I exec into the prometheus container and cat out the generated config file, only the url is there with a remote_timeout setting.

Anyone run into this yet?

@brancz

This comment has been minimized.

Show comment
Hide comment
@brancz

brancz Aug 13, 2018

Member

@incognick there were recently some refactorings on basic auth secret retrieval. Could you open a new issue please?

Member

brancz commented Aug 13, 2018

@incognick there were recently some refactorings on basic auth secret retrieval. Could you open a new issue please?

@incognick

This comment has been minimized.

Show comment
Hide comment
@incognick

incognick Aug 13, 2018

@brancz Thanks! #1788

I didn't look into basic auth too much but it didn't look like that was being passed either.

incognick commented Aug 13, 2018

@brancz Thanks! #1788

I didn't look into basic auth too much but it didn't look like that was being passed either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment