Allow drop_chunks on raw data while keeping continuous aggregate data #1589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cevian
force-pushed
the
cont_agg_data_retention
branch
2 times, most recently
from
afd275d
to
81844f8
Compare
Codecov Report
@@ Coverage Diff @@
## master #1589 +/- ##
==========================================
+ Coverage 91.58% 91.87% +0.29%
==========================================
Files 144 142 -2
Lines 21684 20883 -801
==========================================
- Hits 19859 19186 -673
+ Misses 1825 1697 -128
Continue to review full report at Codecov.
|
cevian
changed the title
cevian
requested review from
gayyappan,
JLockerman,
erimatnor,
k-rus,
mkindahl,
pmwkaa and
WireBaron
cevian
force-pushed
the
cont_agg_data_retention
branch
2 times, most recently
from
472721e
to
cdaac1a
Compare
gayyappan
reviewed
Dec 18, 2019
|
@gayyappan When we move the completion_threshold up (materialize forward in time) we have to use multiple transaction so as to not block inserts for too long because we have to move the invalidation_threshold up. That's why we use multiple transactions to begin with. Thankfully here we only need to process invalidations, not move forward in time and so we can use transactional semantics. If you look at the code we only allow using transactional semantics in this case. |
cevian
force-pushed
the
cont_agg_data_retention
branch
from
cdaac1a
to
e954b00
Compare
JLockerman
reviewed
Dec 19, 2019
| ts_time_bucket_by_type(bucket_width, invalidate_prior_to_time, type); | ||
| /* the last bucket should cover prior_to_time */ | ||
| if (prior_to_time_bucketed != invalidate_prior_to_time) | ||
| prior_to_time_bucketed = int64_saturating_add(prior_to_time_bucketed, bucket_width); |
There was a problem hiding this comment.
do we validate the invalidate_prior_to_time < completed_threshold?
JLockerman
reviewed
Dec 19, 2019
| @@ -620,12 +649,33 @@ materialization_invalidation_log_get_range(int32 materialization_id, Oid type, i | |||
| if (entry_range.start >= completed_threshold) | |||
There was a problem hiding this comment.
should we also have if (entry_range.start >= invalidate_prior_to_time) break ?
JLockerman
reviewed
Dec 19, 2019
gayyappan
reviewed
Dec 19, 2019
| { | ||
| /* mark as not dropped and recreate the pg objects */ | ||
| ScanIterator iterator = | ||
| ts_scan_iterator_create(CHUNK, RowExclusiveLock, CurrentMemoryContext); |
There was a problem hiding this comment.
when is the lock on the row released? we should be holding it till the pg object creation is complete.
There was a problem hiding this comment.
should we add an assert that the chunk is not a compressed chunk? Or should it be applicable to compressed chunks as well?
There was a problem hiding this comment.
we should test this scenario: 1) drop chunk on table 2) change hypertable's chunk interval 3) create some new chunks 4) now try to recreate the old chunk and insert/update/delete/select from the old chunk that was recreated with the old settings.
There was a problem hiding this comment.
The lock on line 716 is not released until end of txn. (see comment above 716). This should work on compressed chunks too. I have a test for this but will add the chunk_interval change.
There was a problem hiding this comment.
oh wait no it shouldn't be allowed on compressed chunks. I'll fix.
There was a problem hiding this comment.
Oh the compressed chunk thing is checked during drop.
There was a problem hiding this comment.
referring to the lock on line 751. RowExclusiveLock on CHUNK. when does that get released?
There was a problem hiding this comment.
The lock is on line 742 LockRelationOid(ht->main_table_relid, ShareUpdateExclusiveLock); Released only at end of txn
| @@ -2151,23 +2266,143 @@ ts_chunk_drop(Chunk *chunk, DropBehavior behavior, int32 log_level) | |||
| chunk->fd.table_name.data); | |||
|
|
|||
| /* Remove the chunk from the hypertable table */ | |||
| ts_chunk_delete_by_relid(chunk->table_id, behavior); | |||
| ts_chunk_delete_by_relid(chunk->table_id, behavior, preserve_catalog_row); | |||
There was a problem hiding this comment.
need a concurrent test where 1) 1 txn drops a chunk 2) the other txn is trying to recreate the chunk. this is to check that we are holding all locks till the first txn completes.
There was a problem hiding this comment.
The lock on line 742 LockRelationOid(ht->main_table_relid, ShareUpdateExclusiveLock); prevents this and hasn't changed with this PR so I don't think we need more tests here.
JLockerman
reviewed
Dec 19, 2019
gayyappan
reviewed
Dec 20, 2019
Allow dropping raw chunks on the raw hypertable while keeping the continuous aggregate. This allows for downsampling data and allows users to save on TCO. We only allow dropping such data when the dropped data is older than the `ignore_invalidation_older_than` parameter on all the associated continuous aggs. This ensures that any modifications to the region of data which was dropped should never be reflected in the continuous agg and thus avoids semantic ambiguity if chunks are dropped but then again recreated due to an insert. Before we drop a chunk we need to make sure to process any continuous aggregate invalidations that were registed on data inside the chunk. Thus we add an option to materialization to perform materialization transactionally, to only process invalidations, and to process invalidation only before a timestamp. We fix drop_chunks and policy to properly process `cascade_to_materialization` as a tri-state variable (unknown, true, false); Existing policy rows should change false to NULL (unknown) and true stays as true since it was explicitly set. Remove the form data for bgw_policy_drop_chunk because there is no good way to represent the tri-state variable in the form data. When dropping chunks with cascade_to_materialization = false, all invalidations on the chunks are processed before dropping the chunk. If we are so far behind that even the completion threshold is inside the chunks being dropped, we error. There are 2 reasons that we error: 1) We can't safely process new ranges transactionally without taking heavy weight locks and potentially locking the entire sytem 2) If a completion threshold is that far behind the system probably has some serious issues anyway.
cevian
force-pushed
the
cont_agg_data_retention
branch
from
e954b00
to
7870d42
Compare
If a chunk is dropped but it has a continuous aggregate that is not dropped we want to preserve the chunk catalog row instead of deleting the row. This is to prevent dangling identifiers in the materialization hypertable. It also preserves the dimension slice and chunk constraints rows for the chunk since those will be necessary when enabling this with multinode and is necessary to recreate the chunk too. The postgres objects associated with the chunk are all dropped (table, constraints, indexes). If data is ever reinserted to the same data region, the chunk is recreated with the same dimension definitions as before. The postgres objects are simply recreated.
Add test for changing chunk interval when recreating chunks. Fix tests for pg10 and pg9.6.
cevian
force-pushed
the
cont_agg_data_retention
branch
from
7870d42
to
a31997b
Compare
JLockerman
approved these changes
Dec 27, 2019
gayyappan
approved these changes
Dec 29, 2019
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 15, 2020
This release adds major new features and bugfixes since the 1.5.1 release. We deem it moderate priority for upgrading. The major new feature in this release allows users to keep the aggregated data in a continuous aggregate while dropping the raw data with drop_chunks. This allows users to save storage by keeping only the aggregates. The semantics of the refresh_lag parameter for continuous aggregates has been changed to be relative to the current timestamp instead of the maximum value in the table. This change requires that an integer_now func be set on hypertables with integer-based time columns to use continuous aggregates on this table. We added a timescaledb.ignore_invalidation_older_than parameter for continuous aggregatess. This parameter accept a time-interval (e.g. 1 month). if set, it limits the amount of time for which to process invalidation. Thus, if timescaledb.ignore_invalidation_older_than = '1 month', then any modifications for data older than 1 month from the current timestamp at modification time may not cause continuous aggregate to be updated. This limits the amount of work that a backfill can trigger. By default, all invalidations are processed. **Major Features** * timescale#1589 Allow drop_chunks while keeping continuous aggregates **Minor Features** * timescale#1568 Add ignore_invalidation_older_than option to continuous aggs * timescale#1575 Reorder group-by clause for continuous aggregates * timescale#1592 Improve continuous agg user messages **Bugfixes** * timescale#1565 Fix partial select query for continuous aggregate * timescale#1591 Fix locf treat_null_as_missing option * timescale#1594 Fix error in compression constraint check * timescale#1603 Add join info to compressed chunk * timescale#1606 Fix constify params during runtime exclusion * timescale#1607 Delete compression policy when drop hypertable * timescale#1608 Add jobs to timescaledb_information.policy_stats * timescale#1609 Fix bug with parent table in decompression **Thanks** * @optijon for reporting an issue with locf treat_null_as_missing option * @acarrera42 for reporting an issue with constify params during runtime exclusion * @ChristopherZellermann for reporting an issue with the compression constraint check * @SimonDelamare for reporting an issue with joining hypertables with compression
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 15, 2020
This release adds major new features and bugfixes since the 1.5.1 release. We deem it moderate priority for upgrading. The major new feature in this release allows users to keep the aggregated data in a continuous aggregate while dropping the raw data with drop_chunks. This allows users to save storage by keeping only the aggregates. The semantics of the refresh_lag parameter for continuous aggregates has been changed to be relative to the current timestamp instead of the maximum value in the table. This change requires that an integer_now func be set on hypertables with integer-based time columns to use continuous aggregates on this table. We added a timescaledb.ignore_invalidation_older_than parameter for continuous aggregatess. This parameter accept a time-interval (e.g. 1 month). if set, it limits the amount of time for which to process invalidation. Thus, if timescaledb.ignore_invalidation_older_than = '1 month', then any modifications for data older than 1 month from the current timestamp at modification time may not cause continuous aggregate to be updated. This limits the amount of work that a backfill can trigger. By default, all invalidations are processed. **Major Features** * timescale#1589 Allow drop_chunks while keeping continuous aggregates **Minor Features** * timescale#1568 Add ignore_invalidation_older_than option to continuous aggs * timescale#1575 Reorder group-by clause for continuous aggregates * timescale#1592 Improve continuous agg user messages **Bugfixes** * timescale#1565 Fix partial select query for continuous aggregate * timescale#1591 Fix locf treat_null_as_missing option * timescale#1594 Fix error in compression constraint check * timescale#1603 Add join info to compressed chunk * timescale#1606 Fix constify params during runtime exclusion * timescale#1607 Delete compression policy when drop hypertable * timescale#1608 Add jobs to timescaledb_information.policy_stats * timescale#1609 Fix bug with parent table in decompression * timescale#1624 Fix drop_chunks for ApacheOnly * timescale#1632 Check for NULL before dereferencing variable **Thanks** * @optijon for reporting an issue with locf treat_null_as_missing option * @acarrera42 for reporting an issue with constify params during runtime exclusion * @ChristopherZellermann for reporting an issue with the compression constraint check * @SimonDelamare for reporting an issue with joining hypertables with compression
svenklemm
added a commit
that referenced
this pull request
Jan 15, 2020
This release adds major new features and bugfixes since the 1.5.1 release. We deem it moderate priority for upgrading. The major new feature in this release allows users to keep the aggregated data in a continuous aggregate while dropping the raw data with drop_chunks. This allows users to save storage by keeping only the aggregates. The semantics of the refresh_lag parameter for continuous aggregates has been changed to be relative to the current timestamp instead of the maximum value in the table. This change requires that an integer_now func be set on hypertables with integer-based time columns to use continuous aggregates on this table. We added a timescaledb.ignore_invalidation_older_than parameter for continuous aggregatess. This parameter accept a time-interval (e.g. 1 month). if set, it limits the amount of time for which to process invalidation. Thus, if timescaledb.ignore_invalidation_older_than = '1 month', then any modifications for data older than 1 month from the current timestamp at modification time may not cause continuous aggregate to be updated. This limits the amount of work that a backfill can trigger. By default, all invalidations are processed. **Major Features** * #1589 Allow drop_chunks while keeping continuous aggregates **Minor Features** * #1568 Add ignore_invalidation_older_than option to continuous aggs * #1575 Reorder group-by clause for continuous aggregates * #1592 Improve continuous agg user messages **Bugfixes** * #1565 Fix partial select query for continuous aggregate * #1591 Fix locf treat_null_as_missing option * #1594 Fix error in compression constraint check * #1603 Add join info to compressed chunk * #1606 Fix constify params during runtime exclusion * #1607 Delete compression policy when drop hypertable * #1608 Add jobs to timescaledb_information.policy_stats * #1609 Fix bug with parent table in decompression * #1624 Fix drop_chunks for ApacheOnly * #1632 Check for NULL before dereferencing variable **Thanks** * @optijon for reporting an issue with locf treat_null_as_missing option * @acarrera42 for reporting an issue with constify params during runtime exclusion * @ChristopherZellermann for reporting an issue with the compression constraint check * @SimonDelamare for reporting an issue with joining hypertables with compression
svenklemm
added a commit
that referenced
this pull request
Jan 15, 2020
This release adds major new features and bugfixes since the 1.5.1 release. We deem it moderate priority for upgrading. The major new feature in this release allows users to keep the aggregated data in a continuous aggregate while dropping the raw data with drop_chunks. This allows users to save storage by keeping only the aggregates. The semantics of the refresh_lag parameter for continuous aggregates has been changed to be relative to the current timestamp instead of the maximum value in the table. This change requires that an integer_now func be set on hypertables with integer-based time columns to use continuous aggregates on this table. We added a timescaledb.ignore_invalidation_older_than parameter for continuous aggregatess. This parameter accept a time-interval (e.g. 1 month). if set, it limits the amount of time for which to process invalidation. Thus, if timescaledb.ignore_invalidation_older_than = '1 month', then any modifications for data older than 1 month from the current timestamp at modification time may not cause continuous aggregate to be updated. This limits the amount of work that a backfill can trigger. By default, all invalidations are processed. **Major Features** * #1589 Allow drop_chunks while keeping continuous aggregates **Minor Features** * #1568 Add ignore_invalidation_older_than option to continuous aggs * #1575 Reorder group-by clause for continuous aggregates * #1592 Improve continuous agg user messages **Bugfixes** * #1565 Fix partial select query for continuous aggregate * #1591 Fix locf treat_null_as_missing option * #1594 Fix error in compression constraint check * #1603 Add join info to compressed chunk * #1606 Fix constify params during runtime exclusion * #1607 Delete compression policy when drop hypertable * #1608 Add jobs to timescaledb_information.policy_stats * #1609 Fix bug with parent table in decompression * #1624 Fix drop_chunks for ApacheOnly * #1632 Check for NULL before dereferencing variable **Thanks** * @optijon for reporting an issue with locf treat_null_as_missing option * @acarrera42 for reporting an issue with constify params during runtime exclusion * @ChristopherZellermann for reporting an issue with the compression constraint check * @SimonDelamare for reporting an issue with joining hypertables with compression
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Detailed description in the commit messages.