New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate Continuous Aggregates to the new format #4552
Migrate Continuous Aggregates to the new format #4552
Conversation
dc84aec
to
3d21c00
Compare
Codecov Report
@@ Coverage Diff @@
## main #4552 +/- ##
==========================================
- Coverage 90.21% 90.21% -0.01%
==========================================
Files 222 222
Lines 41079 41117 +38
==========================================
+ Hits 37061 37092 +31
- Misses 4018 4025 +7
Continue to review full report at Codecov.
|
a523b48
to
0c5dfc3
Compare
0c5dfc3
to
7d71ff8
Compare
ae907c7
to
a6f3f85
Compare
This release adds major new features since the 2.7.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * time_bucket now supports bucketing by month, year and timezone * Improve performance of bulk SELECT and COPY for distributed hypertables * 1 step CAgg policy management * Migrate Continuous Aggregates to the new format **Features** * timescale#4188 Use COPY protocol in row-by-row fetcher * timescale#4307 Mark partialize_agg as parallel safe * timescale#4380 Enable chunk exclusion for space dimensions in UPDATE/DELETE * timescale#4384 Add schedule_interval to policies * timescale#4390 Faster lookup of chunks by point * timescale#4393 Support intervals with day component when constifying now() * timescale#4397 Support intervals with month component when constifying now() * timescale#4405 Support ON CONFLICT ON CONSTRAINT for hypertables * timescale#4412 Add telemetry about replication * timescale#4415 Drop remote data when detaching data node * timescale#4416 Handle TRUNCATE TABLE on chunks * timescale#4425 Add parameter check_config to alter_job * timescale#4430 Create index on Continuous Aggregates * timescale#4439 Allow ORDER BY on continuous aggregates * timescale#4443 Add stateful partition mappings * timescale#4484 Use non-blocking data node connections for COPY * timescale#4495 Support add_dimension() with existing data * timescale#4502 Add chunks to baserel cache on chunk exclusion * timescale#4545 Add hypertable distributed argument and defaults * timescale#4552 Migrate Continuous Aggregates to the new format * timescale#4556 Add runtime exclusion for hypertables * timescale#4561 Change get_git_commit to return full commit hash * timescale#4563 1 step CAgg policy management * timescale#4641 Allow bucketing by month, year, century in time_bucket and time_bucket_gapfill * timescale#4642 Add timezone support to time_bucket **Bugfixes** * timescale#4359 Create composite index on segmentby columns * timescale#4374 Remove constified now() constraints from plan * timescale#4416 Handle TRUNCATE TABLE on chunks * timescale#4478 Synchronize chunk cache sizes * timescale#4486 Adding boolean column with default value doesn't work on compressed table * timescale#4512 Fix unaligned pointer access * timescale#4519 Throw better error message on incompatible row fetcher settings * timescale#4549 Fix dump_meta_data for windows * timescale#4553 Fix timescaledb_post_restore GUC handling * timescale#4573 Load TSL library on compressed_data_out call * timescale#4575 Fix use of `get_partition_hash` and `get_partition_for_key` inside an IMMUTABLE function * timescale#4577 Fix segfaults in compression code with corrupt data * timescale#4580 Handle default privileges on CAggs properly * timescale#4582 Fix assertion in GRANT .. ON ALL TABLES IN SCHEMA * timescale#4583 Fix partitioning functions * timescale#4589 Fix rename for distributed hypertable * timescale#4601 Reset compression sequence when group resets * timescale#4611 Fix a potential OOM when loading large data sets into a hypertable * timescale#4624 Fix heap buffer overflow * timescale#4627 Fix telemetry initialization * timescale#4631 Ensure TSL library is loaded on database upgrades * timescale#4646 Fix time_bucket_ng origin handling * timescale#4647 Fix the error "SubPlan found with no parent plan" that occurred if using joins in RETURNING clause. **Thanks** * @AlmiS for reporting error on `get_partition_hash` executed inside an IMMUTABLE function * @Creatation for reporting an issue with renaming hypertables * @janko for reporting an issue when adding bool column with default value to compressed hypertable * @jayadevanm for reporting error of TRUNCATE TABLE on compressed chunk * @michaelkitson for reporting permission errors using default privileges on Continuous Aggregates * @mwahlhuetter for reporting error in joins in RETURNING clause * @ninjaltd and @mrksngl for reporting a potential OOM when loading large data sets into a hypertable * @PBudmark for reporting an issue with dump_meta_data.sql on Windows * @ssmoss for reporting an issue with time_bucket_ng origin handling
This release adds major new features since the 2.7.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * time_bucket now supports bucketing by month, year and timezone * Improve performance of bulk SELECT and COPY for distributed hypertables * 1 step CAgg policy management * Migrate Continuous Aggregates to the new format **Features** * #4188 Use COPY protocol in row-by-row fetcher * #4307 Mark partialize_agg as parallel safe * #4380 Enable chunk exclusion for space dimensions in UPDATE/DELETE * #4384 Add schedule_interval to policies * #4390 Faster lookup of chunks by point * #4393 Support intervals with day component when constifying now() * #4397 Support intervals with month component when constifying now() * #4405 Support ON CONFLICT ON CONSTRAINT for hypertables * #4412 Add telemetry about replication * #4415 Drop remote data when detaching data node * #4416 Handle TRUNCATE TABLE on chunks * #4425 Add parameter check_config to alter_job * #4430 Create index on Continuous Aggregates * #4439 Allow ORDER BY on continuous aggregates * #4443 Add stateful partition mappings * #4484 Use non-blocking data node connections for COPY * #4495 Support add_dimension() with existing data * #4502 Add chunks to baserel cache on chunk exclusion * #4545 Add hypertable distributed argument and defaults * #4552 Migrate Continuous Aggregates to the new format * #4556 Add runtime exclusion for hypertables * #4561 Change get_git_commit to return full commit hash * #4563 1 step CAgg policy management * #4641 Allow bucketing by month, year, century in time_bucket and time_bucket_gapfill * #4642 Add timezone support to time_bucket **Bugfixes** * #4359 Create composite index on segmentby columns * #4374 Remove constified now() constraints from plan * #4416 Handle TRUNCATE TABLE on chunks * #4478 Synchronize chunk cache sizes * #4486 Adding boolean column with default value doesn't work on compressed table * #4512 Fix unaligned pointer access * #4519 Throw better error message on incompatible row fetcher settings * #4549 Fix dump_meta_data for windows * #4553 Fix timescaledb_post_restore GUC handling * #4573 Load TSL library on compressed_data_out call * #4575 Fix use of `get_partition_hash` and `get_partition_for_key` inside an IMMUTABLE function * #4577 Fix segfaults in compression code with corrupt data * #4580 Handle default privileges on CAggs properly * #4582 Fix assertion in GRANT .. ON ALL TABLES IN SCHEMA * #4583 Fix partitioning functions * #4589 Fix rename for distributed hypertable * #4601 Reset compression sequence when group resets * #4611 Fix a potential OOM when loading large data sets into a hypertable * #4624 Fix heap buffer overflow * #4627 Fix telemetry initialization * #4631 Ensure TSL library is loaded on database upgrades * #4646 Fix time_bucket_ng origin handling * #4647 Fix the error "SubPlan found with no parent plan" that occurred if using joins in RETURNING clause. **Thanks** * @AlmiS for reporting error on `get_partition_hash` executed inside an IMMUTABLE function * @Creatation for reporting an issue with renaming hypertables * @janko for reporting an issue when adding bool column with default value to compressed hypertable * @jayadevanm for reporting error of TRUNCATE TABLE on compressed chunk * @michaelkitson for reporting permission errors using default privileges on Continuous Aggregates * @mwahlhuetter for reporting error in joins in RETURNING clause * @ninjaltd and @mrksngl for reporting a potential OOM when loading large data sets into a hypertable * @PBudmark for reporting an issue with dump_meta_data.sql on Windows * @ssmoss for reporting an issue with time_bucket_ng origin handling
This release adds major new features since the 2.7.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * time_bucket now supports bucketing by month, year and timezone * Improve performance of bulk SELECT and COPY for distributed hypertables * 1 step CAgg policy management * Migrate Continuous Aggregates to the new format **Features** * #4188 Use COPY protocol in row-by-row fetcher * #4307 Mark partialize_agg as parallel safe * #4380 Enable chunk exclusion for space dimensions in UPDATE/DELETE * #4384 Add schedule_interval to policies * #4390 Faster lookup of chunks by point * #4393 Support intervals with day component when constifying now() * #4397 Support intervals with month component when constifying now() * #4405 Support ON CONFLICT ON CONSTRAINT for hypertables * #4412 Add telemetry about replication * #4415 Drop remote data when detaching data node * #4416 Handle TRUNCATE TABLE on chunks * #4425 Add parameter check_config to alter_job * #4430 Create index on Continuous Aggregates * #4439 Allow ORDER BY on continuous aggregates * #4443 Add stateful partition mappings * #4484 Use non-blocking data node connections for COPY * #4495 Support add_dimension() with existing data * #4502 Add chunks to baserel cache on chunk exclusion * #4545 Add hypertable distributed argument and defaults * #4552 Migrate Continuous Aggregates to the new format * #4556 Add runtime exclusion for hypertables * #4561 Change get_git_commit to return full commit hash * #4563 1 step CAgg policy management * #4641 Allow bucketing by month, year, century in time_bucket and time_bucket_gapfill * #4642 Add timezone support to time_bucket **Bugfixes** * #4359 Create composite index on segmentby columns * #4374 Remove constified now() constraints from plan * #4416 Handle TRUNCATE TABLE on chunks * #4478 Synchronize chunk cache sizes * #4486 Adding boolean column with default value doesn't work on compressed table * #4512 Fix unaligned pointer access * #4519 Throw better error message on incompatible row fetcher settings * #4549 Fix dump_meta_data for windows * #4553 Fix timescaledb_post_restore GUC handling * #4573 Load TSL library on compressed_data_out call * #4575 Fix use of `get_partition_hash` and `get_partition_for_key` inside an IMMUTABLE function * #4577 Fix segfaults in compression code with corrupt data * #4580 Handle default privileges on CAggs properly * #4582 Fix assertion in GRANT .. ON ALL TABLES IN SCHEMA * #4583 Fix partitioning functions * #4589 Fix rename for distributed hypertable * #4601 Reset compression sequence when group resets * #4611 Fix a potential OOM when loading large data sets into a hypertable * #4624 Fix heap buffer overflow * #4627 Fix telemetry initialization * #4631 Ensure TSL library is loaded on database upgrades * #4646 Fix time_bucket_ng origin handling * #4647 Fix the error "SubPlan found with no parent plan" that occurred if using joins in RETURNING clause. **Thanks** * @AlmiS for reporting error on `get_partition_hash` executed inside an IMMUTABLE function * @Creatation for reporting an issue with renaming hypertables * @janko for reporting an issue when adding bool column with default value to compressed hypertable * @jayadevanm for reporting error of TRUNCATE TABLE on compressed chunk * @michaelkitson for reporting permission errors using default privileges on Continuous Aggregates * @mwahlhuetter for reporting error in joins in RETURNING clause * @ninjaltd and @mrksngl for reporting a potential OOM when loading large data sets into a hypertable * @PBudmark for reporting an issue with dump_meta_data.sql on Windows * @ssmoss for reporting an issue with time_bucket_ng origin handling
This release adds major new features since the 2.7.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * time_bucket now supports bucketing by month, year and timezone * Improve performance of bulk SELECT and COPY for distributed hypertables * 1 step CAgg policy management * Migrate Continuous Aggregates to the new format **Features** * #4188 Use COPY protocol in row-by-row fetcher * #4307 Mark partialize_agg as parallel safe * #4380 Enable chunk exclusion for space dimensions in UPDATE/DELETE * #4384 Add schedule_interval to policies * #4390 Faster lookup of chunks by point * #4393 Support intervals with day component when constifying now() * #4397 Support intervals with month component when constifying now() * #4405 Support ON CONFLICT ON CONSTRAINT for hypertables * #4412 Add telemetry about replication * #4415 Drop remote data when detaching data node * #4416 Handle TRUNCATE TABLE on chunks * #4425 Add parameter check_config to alter_job * #4430 Create index on Continuous Aggregates * #4439 Allow ORDER BY on continuous aggregates * #4443 Add stateful partition mappings * #4484 Use non-blocking data node connections for COPY * #4495 Support add_dimension() with existing data * #4502 Add chunks to baserel cache on chunk exclusion * #4545 Add hypertable distributed argument and defaults * #4552 Migrate Continuous Aggregates to the new format * #4556 Add runtime exclusion for hypertables * #4561 Change get_git_commit to return full commit hash * #4563 1 step CAgg policy management * #4641 Allow bucketing by month, year, century in time_bucket and time_bucket_gapfill * #4642 Add timezone support to time_bucket **Bugfixes** * #4359 Create composite index on segmentby columns * #4374 Remove constified now() constraints from plan * #4416 Handle TRUNCATE TABLE on chunks * #4478 Synchronize chunk cache sizes * #4486 Adding boolean column with default value doesn't work on compressed table * #4512 Fix unaligned pointer access * #4519 Throw better error message on incompatible row fetcher settings * #4549 Fix dump_meta_data for windows * #4553 Fix timescaledb_post_restore GUC handling * #4573 Load TSL library on compressed_data_out call * #4575 Fix use of `get_partition_hash` and `get_partition_for_key` inside an IMMUTABLE function * #4577 Fix segfaults in compression code with corrupt data * #4580 Handle default privileges on CAggs properly * #4582 Fix assertion in GRANT .. ON ALL TABLES IN SCHEMA * #4583 Fix partitioning functions * #4589 Fix rename for distributed hypertable * #4601 Reset compression sequence when group resets * #4611 Fix a potential OOM when loading large data sets into a hypertable * #4624 Fix heap buffer overflow * #4627 Fix telemetry initialization * #4631 Ensure TSL library is loaded on database upgrades * #4646 Fix time_bucket_ng origin handling * #4647 Fix the error "SubPlan found with no parent plan" that occurred if using joins in RETURNING clause. **Thanks** * @AlmiS for reporting error on `get_partition_hash` executed inside an IMMUTABLE function * @Creatation for reporting an issue with renaming hypertables * @janko for reporting an issue when adding bool column with default value to compressed hypertable * @jayadevanm for reporting error of TRUNCATE TABLE on compressed chunk * @michaelkitson for reporting permission errors using default privileges on Continuous Aggregates * @mwahlhuetter for reporting error in joins in RETURNING clause * @ninjaltd and @mrksngl for reporting a potential OOM when loading large data sets into a hypertable * @PBudmark for reporting an issue with dump_meta_data.sql on Windows * @ssmoss for reporting an issue with time_bucket_ng origin handling
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (timescale#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Timescale 2.8 released a migration path from the old format of Continuous Aggregate to the new format (#4552). Unfortunately it lacks of proper tests when a non-superuser execute the migration. For a non-superuser execute the migration properly it requires SELECT/INSERT/UPDATE permissions in some catalog objects: * _timescaledb_catalog.continuous_agg_migrate_plan * _timescaledb_catalog.continuous_agg_migrate_plan_step * _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq Improved the regression tests to cover the lack of permissions in the catalog objects for non-superusers.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (timescale#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (timescale#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Recently we fixed a DDL error (#4739) after upgrading to 2.8.0 version that surprisly the CI upgrade/dowgrade tests didn't complained during the development of the feature (#4552). Fixed it by adding an specific query in the `post.catalog.sql` script to make sure we'll check all the constraints of our internal tables and catalog.
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that store the final aggregation state instead of the byte array of
the partial aggregate state, offering multiple opportunities of
optimizations as well a more compact form.
When upgrading to Timescale 2.7, new created Continuous Aggregates
are using the new format, but existing Continuous Aggregates keep
using the format they were defined with.
Created a procedure to upgrade existing Continuous Aggregates from
the old format to the new format, by calling a simple procedure:
Closes #4424