Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always validate existing database and extension #2772

Merged
merged 1 commit into from Dec 29, 2020

Conversation

erimatnor
Copy link
Contributor

This change ensures the database and extension is validated whenever
these objects aren't created, instead of only doing validation when
bootstrap=>false is passed when adding a data node.

This fixes a corner case where a data node could be added and removed
several times, even though the data node's database was already marked
as having been part of a multi-node setup.

@erimatnor erimatnor force-pushed the fix-data-node-validation branch 2 times, most recently from e99c362 to 43ad486 Compare December 22, 2020 13:47
@erimatnor erimatnor changed the title Validate database and extension during bootstrap Always validate existing database and extension Dec 22, 2020
@codecov
Copy link

codecov bot commented Dec 22, 2020

Codecov Report

Merging #2772 (fce26f6) into master (71633c3) will increase coverage by 0.10%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2772      +/-   ##
==========================================
+ Coverage   90.11%   90.22%   +0.10%     
==========================================
  Files         212      212              
  Lines       34665    34617      -48     
==========================================
- Hits        31239    31232       -7     
+ Misses       3426     3385      -41     
Impacted Files Coverage Δ
tsl/src/data_node.c 96.08% <100.00%> (-0.18%) ⬇️
src/loader/bgw_message_queue.c 84.51% <0.00%> (-3.23%) ⬇️
tsl/src/dist_util.c 91.94% <0.00%> (-1.35%) ⬇️
tsl/src/data_node_dispatch.c 97.02% <0.00%> (-0.25%) ⬇️
src/import/planner.c 70.30% <0.00%> (+11.12%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71633c3...fce26f6. Read the comment docs.

@erimatnor erimatnor marked this pull request as ready for review December 22, 2020 14:59
@erimatnor erimatnor requested a review from a team as a code owner December 22, 2020 14:59
Copy link
Contributor

@k-rus k-rus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest to merge the commits into one, since the test commit checks that the non-bootrstrap option works and will fail without the first commit. I hope I haven't missed anything in the logic.

@erimatnor
Copy link
Contributor Author

erimatnor commented Dec 22, 2020

I suggest to merge the commits into one, since the test commit checks that the non-bootrstrap option works and will fail without the first commit. I hope I haven't missed anything in the logic.

All tests pass in both commits, independently. The second commit adds a test case that covers the case that didn't work as expected. Although I can certainly squash this, I personally don't see any issues with separating changes into logical units.

@k-rus
Copy link
Contributor

k-rus commented Dec 23, 2020

I suggest to merge the commits into one, since the test commit checks that the non-bootrstrap option works and will fail without the first commit. I hope I haven't missed anything in the logic.

All tests pass in both commits, independently. The second commit adds a test case that covers the case that didn't work as expected. Although I can certainly squash this, I personally don't see any issues with separating changes into logical units.

The first commit implements validation of database and extension if bootstrap is false. However, I see no tests in the this commit, which checks this. It is good practice to have test in the same commit as implementation for several reasons:

  1. It helps to understand the commit better.
  2. It allows to cherry pick or rollback the implementation with the regression test together.

If I understood the second commit correctly, it includes a test for the first commit. Correct me if I am wrong and test is missing.

This change ensures the database and extension is validated whenever
these objects aren't created, instead of only doing validation when
`bootstrap=>false` is passed when adding a data node.

This fixes a corner case where a data node could be added and removed
several times, even though the data node's database was already marked
as having been part of a multi-node setup.

A new test checks that a data node cannot be re-added after deleting
it on the access node, irrespective of whether one bootstraps the data
node or not when it is added.
@erimatnor erimatnor merged commit c224bc7 into timescale:master Dec 29, 2020
@erimatnor erimatnor deleted the fix-data-node-validation branch December 29, 2020 12:37
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when seting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script

**Thanks**
* @sgorsh for reporting an issue when using pgAdmin on windows
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @PhilippJust for reporting an issue with add_job and initial_start
* @alex88 for reporting an issue with joined hypertables
@svenklemm svenklemm mentioned this pull request Jan 27, 2021
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when seting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when seting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when seting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 27, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when seting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when setting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node
* timescale#2866 Avoid partitionwise planning of partialize_agg
* timescale#2868 Fix corruption in gapfill plan
* timescale#2874 Fix partitionwise agg crash due to uninitialized memory

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
  able to find pathkey item to sort
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when setting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node
* timescale#2866 Avoid partitionwise planning of partialize_agg
* timescale#2868 Fix corruption in gapfill plan
* timescale#2874 Fix partitionwise agg crash due to uninitialized memory

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
  able to find pathkey item to sort
svenklemm added a commit to svenklemm/timescaledb that referenced this pull request Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* timescale#2772 Always validate existing database and extension
* timescale#2780 Fix config enum entries for remote data fetcher
* timescale#2806 Add check for dropped chunk on update
* timescale#2828 Improve cagg watermark caching
* timescale#2842 Do not mark job as started when setting next_start field
* timescale#2845 Fix continuous aggregate privileges during upgrade
* timescale#2851 Fix nested loop joins that involve compressed chunks
* timescale#2860 Fix projection in ChunkAppend nodes
* timescale#2861 Remove compression stat update from update script
* timescale#2865 Apply volatile function quals at decompresschunk node
* timescale#2866 Avoid partitionwise planning of partialize_agg
* timescale#2868 Fix corruption in gapfill plan
* timescale#2874 Fix partitionwise agg crash due to uninitialized memory

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
  able to find pathkey item to sort
svenklemm added a commit that referenced this pull request Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* #2772 Always validate existing database and extension
* #2780 Fix config enum entries for remote data fetcher
* #2806 Add check for dropped chunk on update
* #2828 Improve cagg watermark caching
* #2842 Do not mark job as started when setting next_start field
* #2845 Fix continuous aggregate privileges during upgrade
* #2851 Fix nested loop joins that involve compressed chunks
* #2860 Fix projection in ChunkAppend nodes
* #2861 Remove compression stat update from update script
* #2865 Apply volatile function quals at decompresschunk node
* #2866 Avoid partitionwise planning of partialize_agg
* #2868 Fix corruption in gapfill plan
* #2874 Fix partitionwise agg crash due to uninitialized memory

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
  able to find pathkey item to sort
svenklemm added a commit that referenced this pull request Jan 28, 2021
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.

In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.

**Bugfixes**
* #2772 Always validate existing database and extension
* #2780 Fix config enum entries for remote data fetcher
* #2806 Add check for dropped chunk on update
* #2828 Improve cagg watermark caching
* #2838 Fix catalog repair in update script
* #2842 Do not mark job as started when setting next_start field
* #2845 Fix continuous aggregate privileges during upgrade
* #2851 Fix nested loop joins that involve compressed chunks
* #2860 Fix projection in ChunkAppend nodes
* #2861 Remove compression stat update from update script
* #2865 Apply volatile function quals at decompresschunk node
* #2866 Avoid partitionwise planning of partialize_agg
* #2868 Fix corruption in gapfill plan
* #2874 Fix partitionwise agg crash due to uninitialized memory

**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
  able to find pathkey item to sort
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants