Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove completed threshold in continuous aggregate materializer #2178

Closed
erimatnor opened this issue Jul 31, 2020 · 0 comments · Fixed by #2382
Closed

Remove completed threshold in continuous aggregate materializer #2178

erimatnor opened this issue Jul 31, 2020 · 0 comments · Fixed by #2382
Assignees
Milestone

Comments

@erimatnor
Copy link
Contributor

erimatnor commented Jul 31, 2020

The new refresh framework for continuous aggregates removes the need for the completed threshold. Progress in materialization will instead be tracked only in the cagg invalidation log.

Tracking materialization progress without the completed threshold necessitates initializing the cagg invalidation log with an infinite invalidation entry: [ -Infinity, + Infinity].

Adding such an entry allows tracking exactly the regions that are invalid and by inverse the regions that are valid. The information that was tracked by the old completed threshold will exist as the entry [ X, +Infinity ] in the cagg invalidation log. The completed threshold table can be removed.

However, with the new refresh framework, the completed threshold doesn't have much purpose as there is no distinction made between invalid and previously not materialized regions. Instead, a background policy an be free to decide, using heuristics, which regions within a refresh window to prioritize during each refresh.

@erimatnor erimatnor added this to the 2.0.0 milestone Jul 31, 2020
@erimatnor erimatnor self-assigned this Jul 31, 2020
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 14, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 14, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 14, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit to erimatnor/timescaledb that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes timescale#2178
erimatnor added a commit that referenced this issue Sep 15, 2020
The completed threshold in the TimescaleDB catalog is no longer used
by the refactored continuous aggregates, so it is removed.

Fixes #2178
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant