Remote storage

[juliusv]

Prometheus needs to be able to interface with a remote and scalable data store for long-term storage/retrieval.

[johann8384]

Is there anyone planning to work on this? Is the work done in the opentsdb-integration branch still valid or has the rest of the code-base moved past that?

[beorn7]

The opentsdb-integration branch is indeed completely outdated (still using the old storage backend etc.). Personally, I'm a great fan of the OpenTSDB integration, but where I work, there is not an urgent enough requirement to justify a high priority from my side...

[juliusv]

To be clear, the outdated "opentsdb-integration" was only for the
proof-of-concept read-back support (querying OpenTSDB through Prometheus).

Writing into OpenTSDB should be experimentally supported in master, but
the last time we tried it was a year ago on a single-node OpenTSDB.

You initially asked on #10:

"I added the storage.remote.url command line flag, but as far as I can tell
Prometheus doesn't attempt to store any metrics there."

A couple of questions:

• did you enable the OpenTSDB option "tsd.core.auto_create_metrics"?
  Otherwise OpenTSDB won't auto-create metrics for you, as the option is
  false by default. See
  http://opentsdb.net/docs/build/html/user_guide/configuration.html
• if you run Prometheus with -logtostderr, do you see any relevant log
  output? If there is an error sending samples to TSDB, it should be logged
  (glog.Warningf("error sending %d samples to TSDB: %s", len(s), err))
• Prometheus also exports metrics itself about sending to OpenTSDB. On
  /metrics of your Prometheus server, you should find the counter metrics
  "prometheus_remote_storage_sent_errors_total" and
  "prometheus_remote_storage_sent_samples_total". What do these say?

Cheers,
Julius

On Thu, Feb 5, 2015 at 9:22 AM, Björn Rabenstein notifications@github.com
wrote:

The opentsdb-integration branch is indeed completely outdated (still using
the old storage backend etc.). Personally, I'm a great fan of the OpenTSDB
integration, but where I work, there is not an urgent enough requirement to
justify a high priority from my side...

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[sammcj]

I cannot +1 this enough

[mwitkow]

Is InfluxDB on the cards in any way? :)

[beorn7]

Radio Yerevan: "In principle yes." (Please forgive that Eastern European digression... ;)

[mwitkow]

:D That was slightly before my time ;)

[juliusv]

See also: https://twitter.com/juliusvolz/status/569509228462931968

We're just waiting for InfluxDB 0.9.0, which has a new data model which
should be more compatible with Prometheus's.

On Thu, Mar 5, 2015 at 10:31 AM, Michal Witkowski notifications@github.com
wrote:

:D That was slightly before my time ;)

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[pires]

We're just waiting for InfluxDB 0.9.0, which has a new data model which
should be more compatible with Prometheus's.

Can I say awesome more than once? Awesome!

[fabxc]

Unfortunately, @juliusv ran some tests with 0.9 and InfluxDB consumed 14x more storage than Prometheus.

Before it was an overhead of 11x but Prometheus's could reduce storage size significantly since then - so in reality InfluxDB has apparently improved in that regard.
Nonetheless, InfluxDB did not turn out to be the eventual answer for long-term storage, yet.

[beorn7]

At least experimental write support is in master, as of today, so anybody can play with Influxdb receiving Prometheus metrics. Quite possible somebody finds the reason for the blow-up in storage space and everything will be unicorns and rainbows in the end...

[pires]

@beorn7 that's great. TBH I'm not concerned about disk space, it's the cheapest resource on the cloud after all. Not to mention, I'm expecting to hold data with a very small TTL, i.e. few weeks.

[beorn7]

@pires In that case, why not just run two identically configured Prometheis with a reasonably large disk?
A few weeks or months is usually fine as retention time for Prometheus. (Default is 15d for a reason... :) The only problem is that if your disk breaks, your data is gone, but for that, you have the other server.

[fabxc]

@pires do you have a particular reason to hold the data in another database for that time? "A few weeks" does not seem to require a long-term storage solution. Prometheus's default retention time is 15 days - increasing that to 30 or even 60 days should not be a problem.