Add check for dropped chunk on update #2806
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mkindahl
force-pushed
the
update_drop_chunk_problem
branch
from
8dd18b7
to
f5db7df
Compare
Codecov Report
@@ Coverage Diff @@
## master #2806 +/- ##
==========================================
+ Coverage 90.10% 90.22% +0.11%
==========================================
Files 212 212
Lines 34670 34620 -50
==========================================
- Hits 31240 31236 -4
+ Misses 3430 3384 -46
Continue to review full report at Codecov.
|
mkindahl
force-pushed
the
update_drop_chunk_problem
branch
6 times, most recently
from
0def106
to
d3bcc57
Compare
If `drop_chunks` has been executed on a hypertable that has a continuous aggregate defined, the chunks will be removed and marked as dropped in `_timescaledb_catalog.chunk` but the lines will not be removed. This will cause problems for the update script since it is missing a check to only process chunks that are not dropped and will try to cast the chunk name into a `REGCLASS` for a table that does not exist. This commit fixes this by adding a check that the chunk is not dropped and also fixes the update test to not count dropped chunks when comparing with the chunk index since the chunk index does not count dropped chunks. Fixes timescale#2791
mkindahl
force-pushed
the
update_drop_chunk_problem
branch
from
d3bcc57
to
f66eac3
Compare
mkindahl
changed the title
mkindahl
requested review from
k-rus,
svenklemm and
gayyappan
and removed request for
a team
svenklemm
approved these changes
Jan 13, 2021
pmwkaa
approved these changes
Jan 13, 2021
| INSERT INTO drop_test | ||
| SELECT | ||
| time, | ||
| (random() * 3 + 1)::int, |
There was a problem hiding this comment.
do we set seed somewhere for that? if not maybe it makes sense to do that before this test case
There was a problem hiding this comment.
Just using this data to generate chunks, so only the number of rows is important.
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when seting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script **Thanks** * @sgorsh for reporting an issue when using pgAdmin on windows * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @PhilippJust for reporting an issue with add_job and initial_start * @alex88 for reporting an issue with joined hypertables
Merged
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when seting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when seting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when seting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when seting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when setting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node * timescale#2866 Avoid partitionwise planning of partialize_agg * timescale#2868 Fix corruption in gapfill plan * timescale#2874 Fix partitionwise agg crash due to uninitialized memory **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @markatosi for reporting a segfault with partitionwise aggregates enabled * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows * @WarriorOfWire for reporting the bug with gapfill queries not being able to find pathkey item to sort
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when setting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node * timescale#2866 Avoid partitionwise planning of partialize_agg * timescale#2868 Fix corruption in gapfill plan * timescale#2874 Fix partitionwise agg crash due to uninitialized memory **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @markatosi for reporting a segfault with partitionwise aggregates enabled * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows * @WarriorOfWire for reporting the bug with gapfill queries not being able to find pathkey item to sort
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * timescale#2772 Always validate existing database and extension * timescale#2780 Fix config enum entries for remote data fetcher * timescale#2806 Add check for dropped chunk on update * timescale#2828 Improve cagg watermark caching * timescale#2842 Do not mark job as started when setting next_start field * timescale#2845 Fix continuous aggregate privileges during upgrade * timescale#2851 Fix nested loop joins that involve compressed chunks * timescale#2860 Fix projection in ChunkAppend nodes * timescale#2861 Remove compression stat update from update script * timescale#2865 Apply volatile function quals at decompresschunk node * timescale#2866 Avoid partitionwise planning of partialize_agg * timescale#2868 Fix corruption in gapfill plan * timescale#2874 Fix partitionwise agg crash due to uninitialized memory **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @markatosi for reporting a segfault with partitionwise aggregates enabled * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows * @WarriorOfWire for reporting the bug with gapfill queries not being able to find pathkey item to sort
svenklemm
added a commit
that referenced
this pull request
Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * #2772 Always validate existing database and extension * #2780 Fix config enum entries for remote data fetcher * #2806 Add check for dropped chunk on update * #2828 Improve cagg watermark caching * #2842 Do not mark job as started when setting next_start field * #2845 Fix continuous aggregate privileges during upgrade * #2851 Fix nested loop joins that involve compressed chunks * #2860 Fix projection in ChunkAppend nodes * #2861 Remove compression stat update from update script * #2865 Apply volatile function quals at decompresschunk node * #2866 Avoid partitionwise planning of partialize_agg * #2868 Fix corruption in gapfill plan * #2874 Fix partitionwise agg crash due to uninitialized memory **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @markatosi for reporting a segfault with partitionwise aggregates enabled * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows * @WarriorOfWire for reporting the bug with gapfill queries not being able to find pathkey item to sort
svenklemm
added a commit
that referenced
this pull request
Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in continuous aggregates, compression, JOINs with hypertables and when upgrading from previous versions. **Bugfixes** * #2772 Always validate existing database and extension * #2780 Fix config enum entries for remote data fetcher * #2806 Add check for dropped chunk on update * #2828 Improve cagg watermark caching * #2838 Fix catalog repair in update script * #2842 Do not mark job as started when setting next_start field * #2845 Fix continuous aggregate privileges during upgrade * #2851 Fix nested loop joins that involve compressed chunks * #2860 Fix projection in ChunkAppend nodes * #2861 Remove compression stat update from update script * #2865 Apply volatile function quals at decompresschunk node * #2866 Avoid partitionwise planning of partialize_agg * #2868 Fix corruption in gapfill plan * #2874 Fix partitionwise agg crash due to uninitialized memory **Thanks** * @alex88 for reporting an issue with joined hypertables * @brian-from-quantrocket for reporting an issue with extension update and dropped chunks * @dhodyn for reporting an issue when joining compressed chunks * @markatosi for reporting a segfault with partitionwise aggregates enabled * @PhilippJust for reporting an issue with add_job and initial_start * @sgorsh for reporting an issue when using pgAdmin on windows * @WarriorOfWire for reporting the bug with gapfill queries not being able to find pathkey item to sort
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If
drop_chunkshas been executed on a hypertable that has acontinuous aggregate defined, the chunks will be removed and marked as
dropped in
_timescaledb_catalog.chunkbut the lines will not beremoved. This will cause problems for the update script since it is
missing a check to only process chunks that are not dropped and will
try to cast the chunk name into a
REGCLASSfor a table that does notexist.
This commit fixes this by adding a check that the chunk is not dropped
and also fixes the update test to not count dropped chunks when
comparing with the chunk index since the chunk index does not count
dropped chunks.
Fixes #2791