Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Backfill for continuous query should happen in chunks #186
The backfill for continuous queries could blow up if there is already a lot of data in the db that matches the query. Do this to make sure it happens in chunks and doesn't explode.
Regex queries should do a query with limit 1 to pull back all the series then fill each one like so:
Find out the first group to file by doing limit 1 order desc. Then do chunks of groups. If it's group by 1d, do each day at a time. Otherwise grab somewhere around an hour or two at a time.
After looking at the code, I saw that continuous queries weren't backfilling at all (at least for the first one created). I fixed that, but they're only backfilling for queries that have a group by interval.
We should keep it this way until the backfill in chunks is done. Then backfill for continuous queries that don't have a group by time.