Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Continuous downsampling #1179

Closed
Sytten opened this issue Apr 20, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@Sytten
Copy link

commented Apr 20, 2019

I know this has been asked a few times already over the years (#686, #242), but I wanted to know if there was an update on the priority of this feature. I feel that almost all time series application need this one way or another.
Maybe the integration with pipelinedb (#14) would could solve this issue?

@mfreed mfreed added the question label Apr 20, 2019

@mfreed

This comment has been minimized.

Copy link
Member

commented Apr 20, 2019

Hi @Sytten: Thanks for the interest. We'll be releasing support for in-database continuous aggregations in v1.3, which should be released within the next few weeks. Code should hit master branch within a few days.

@Sytten

This comment has been minimized.

Copy link
Author

commented Apr 20, 2019

Wow, this is amazing! Do you have a public roadmap where we can the planned features?

@mfreed

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

We have not published it yet. Can you say more about your particular application needs or use case?

@mfreed mfreed added the enhancement label Apr 22, 2019

@mfreed mfreed changed the title Downsampling rules Feature request: Continuous downsampling Apr 22, 2019

@mfreed mfreed removed the question label Apr 22, 2019

@Sytten

This comment has been minimized.

Copy link
Author

commented Apr 22, 2019

Mostly sensor sampling that we want to keep for a longer period without the same accuracy. If we can keep a count and a sum that would be perfect. We were considering using PipelineDB for that, but it is not well supported by managed services.

@nickbabcock

This comment has been minimized.

Copy link

commented Apr 25, 2019

Very neat. I was about to roll my own by having a realtime table that is downsampled (then drop_chunks) every day into 15 minute intervals into another table (another hypertable) and have a view encapsulate both for grafana. But if this will cover that use case, then I'm all for it and can wait! Hopefully this feature is slotted for the community edition. 😄

I'm curious how you handle aggregations. When graphite downsamples data, one can configure if the resulting value is averaged, summed, min value, max value, or last value.

If you want to know more about my use case: I have timescale at the receiving end of hardware sensor data

@mfreed

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@nickbabcock The initial release of continuous aggregations will all be in the community edition. Future extensions may include some enterprise features. You can indeed con figure the result fairly flexibly -- basically any aggregate in PostgreSQL that can be efficiently parallelized (average, sum, min, max, count, etc.).

What's interesting about our implementation is it leverages partials: The aggregation doesn't actually store an average, but instead stores sums & counts, and then will compute the average from those at query time -- all transparent to the user -- which will allow us to do a lot more interesting things in transparently combining pre-computed aggregates (e.g., in a hierarchical fashion).

And here it is: #1184

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.