Add a way to ship 2.0 storage blocks elsewhere

[gouthamve]

I wanted to write a tool that ships the 2.0 blocks to a remote storage, but looks like there is interest from the others as well. Especially as it allows for bulk insert which is much more efficient.

The plan currently for me is: Make a tool (promtool subcommand) which actually ships the block off to the remote-storage and mark it experimental and iterate on the fields and format of the API and once we have a couple of users, fold it into Prometheus.

@tomwilkie @pauldix Do you guys think this makes sense? Or do we want to ship the chunks instead of the block?

Shipping the block actually makes it really hard to write the integration in a language other than Go and will heavily depend on the structure of TSDB.

[pauldix]

I'm not sure if the format should be tied to chunks or blocks. Although if you just send blocks it'll be more efficient since you're just copying over the underlying datastructures untouched. Although it looks like tombstones might cause a problem with this since they'll have to be shipped too?

Overall I think the real goal would be to have a bulk read/write API. I think from the read perspective it would take a start and end time and include data from all series in that range. And the write would take some contiguous block of time to write all in one go. If you think it makes sense to use blocks then I'm +1. We can work with the actual prometheus/tsdb structures or some other intermediate format. Either way we'll have to do a conversions to/from the internal InfluxDB format.

[brian-brazil]

I think 2.0 blocks is what we should use here, and keep this coupled with Prometheus. I think an API for bulk read would be a bad idea, as the response size would be massive and we already have one of those in the form of the query endpoint.

Bulk write is a separate topic.

[pauldix]

@brian-brazil fair enough. If the write API sends blocks, we'll be able to work with it.

[brian-brazil]

@fabxc With your recent change over in TSDB don't we just need to add a paramater to the snapshot API to make this practical?

[brian-brazil]

And that API is now there, so I'm calling this done.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.