Change compression settings to be per chunk #6513

svenklemm · 2024-01-11T09:23:06Z

This patch implements changes to the compressed hypertable to allow per chunk configuration. To enable this the compressed hypertable can no longer be in an inheritance tree as the schema of the compressed chunk is determined by the compression settings. While this patch implements all the underlying infrastructure changes the restrictions for changing compression settings remain intact and will be lifted in a followup patch.

github-actions · 2024-01-11T09:23:23Z

@mkindahl, @nikkhils: please review this pull request.

Powered by pull-review

antekresic

Took a high level pass, the thing to look into is why the plans changed so much.

Using indexes more than before, definitely unexpected. This might uncover some oversight in the current planner implementation or could be due to the fact that we are using chunk indexes instead of the index on the compressed hypertable.

Worth looking into.

tsl/test/expected/compression_ddl.out

tsl/test/expected/compression_hypertable.out

tsl/src/compression/compression_storage.c

tsl/test/expected/compression.out

tsl/src/compression/compression_storage.c

codecov · 2024-01-12T20:48:01Z

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (55ae29c) 79.73% compared to head (a4d3a82) 79.66%.

Files	Patch %	Lines
tsl/src/compression/compression_storage.c	90.00%	3 Missing and 12 partials ⚠️
tsl/src/compression/create.c	80.59%	3 Missing and 10 partials ⚠️
src/process_utility.c	85.00%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6513      +/-   ##
==========================================
- Coverage   79.73%   79.66%   -0.08%     
==========================================
  Files         187      188       +1     
  Lines       36720    36735      +15     
  Branches     9296     9293       -3     
==========================================
- Hits        29280    29266      -14     
- Misses       3103     3128      +25     
- Partials     4337     4341       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

antekresic

LGTM, just a couple minor nits about migration messaging.

antekresic · 2024-01-15T11:52:03Z

sql/updates/reverse-dev.sql

+      RAISE USING
+        ERRCODE = 'feature_not_supported',
+        MESSAGE = 'Cannot downgrade compressed hypertables with no compressed chunks',
+        DETAIL = 'The following hypertable is affected: '|| ht_uncomp::text;


Can we suggest removing compression from the hypertable since there is nothing compressed? That way you don't need to downgrade it.

antekresic · 2024-01-15T11:54:45Z

sql/updates/reverse-dev.sql

+        WHERE relid = hypertable OR relid = ANY(chunk_relids)
+      ) dist_settings HAVING count(*) > 1
+		) THEN
+      RAISE EXCEPTION 'Cannot downgrade hypertables with distinct compression settings';


This might need more details on how to get to a state where downgrade will be possible.

i.e. recompressing chunks to have a single compression setting.

erimatnor

Mostly nits, but my main concern is the update/downgrade script and how it will work when people are doing upgrades on live data with lots of compressed chunks. It seems that locks could be a problem.

erimatnor · 2024-01-16T08:57:27Z

sql/updates/latest-dev.sql

+BEGIN
+  SET timescaledb.restoring TO ON;
+
+  -- Detach compressed chunks from their parent hypertables


I am a bit worried that modifying data chunks during upgrades like this is going to be problematic. I don't know exactly the type of lock taken by NO INHERIT but it is probably a heavy one (maybe even AccessExclusive).

This could lead to major problems for customers that have lots of compressed chunks, especially when this happens during automatic upgrades.

At least I think we should clearly understand the implications and behavior of doing this in the update script.

Since NO INHERIT does not rewrite the table it should finish instantly but it does indeed require AccessExclusive lock. But AccessExclusive locks are required by most non-patch releases I think this only becomes a problem when we try to do expensive data migrations like rewriting tables which isn't the case here.

erimatnor · 2024-01-16T09:13:39Z

tsl/src/compression/create.c

+	row_size += 8;	/* sequence_num */
+	row_size += 4;	/* count */
+	row_size += 16; /* min/max */


We should have a canonical place (header) where we define the metadata columns, including types and sizes. If we hard-code the information like this in every place it i used, it is easy to make a mistake.

erimatnor · 2024-01-16T09:16:33Z

tsl/src/compression/create.c

+				 errmsg("compression cannot be used on table with row security")));
+
+	Relation rel = table_open(ht->main_table_relid, AccessShareLock);
+	TupleDesc tupdesc = rel->rd_att;


Suggested change

TupleDesc tupdesc = rel->rd_att;

TupleDesc tupdesc = RelationGetDescr(rel);

tsl/src/compression/create.c

erimatnor · 2024-01-16T09:29:47Z

tsl/src/compression/compression_storage.c

+static List *get_fk_constraints(Oid reloid);
+
+int32
+compression_hypertable_create(Hypertable *ht, Oid owner, Oid tablespace_oid)


Is this mostly code that has been moved?

Are there any changes here to be on the lookout for?

Perhaps it would be better to do refactoring separately...

Most of this has been moved, most if the code that used to be done on the hypertable is now done per chunk and we no longer rely on inheritance propagation of columns and their attributes but now manually iterate over compressed chunks for these operations.

akuzm · 2024-01-16T15:03:01Z

tsl/src/compression/compression_storage.c

+
+		/* skip system columns */
+		if (col_attr->attnum <= 0)
+			continue;


Can we have system columns in a physical tuple of relation? I think no, they can only appear in scan tuples.

Hmm you might be right that those are skipped already but pg_attribute does include the system columns and we are dealing with Form_pg_attribute here

This patch implements changes to the compressed hypertable to allow per chunk configuration. To enable this the compressed hypertable can no longer be in an inheritance tree as the schema of the compressed chunk is determined by the compression settings. While this patch implements all the underlying infrastructure changes, the restrictions for changing compression settings remain intact and will be lifted in a followup patch.

The commit timescale#6513 removed some restricted chunk operations, enabling adding constraints to OSM chunks directly. This operation is blocked on OSM chunks. The present commit ensures that adding a constraint directly on an OSM chunk is blocked.

The commit #6513 removed some restricted chunk operations, enabling adding constraints to OSM chunks directly. This operation is blocked on OSM chunks. The present commit ensures that adding a constraint directly on an OSM chunk is blocked.

@coney21

This release contains performance improvements, an improved hypertable DDL API and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations (100k limit, configurable) * Helper functions for determining compression settings **Removal notice: Multi-node support** TimescaleDB 2.13 is the last version that includes multi-node support. Multi-node support is effectively removed in 2.14 version. Learn more about it [here](docs/MultiNodeDeprecation.md). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include recompress_chunk procedure. Its function will be replaced by compress_chunk function which should work on uncompressed and partially compressed chunk. It should be used to fully compress all chunks. **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * timescale#6325 Add plan-time chunk exclusion for real-time CAggs * timescale#6360 Remove support for creating Continuous Aggregates with old format * timescale#6386 Add functions for determining compression defaults * timescale#6410 Remove multinode public API * timescale#6440 Allow SQLValueFunction pushdown into compressed scan * timescale#6463 Support approximate hypertable size * timescale#6513 Make compression settings per chunk * timescale#6529 Remove reindex_relation from recompression * timescale#6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * timescale#6545 Remove restrictions for changing compression settings * timescale#6566 Limit tuple decompression during DML operations * timescale#6579 Change compress_chunk and decompress_chunk to idempotent version by default * timescale#6608 Add LWLock for OSM usage in loader * timescale#6609 Deprecate recompress_chunk * timescale#6609 Add optional recompress argument to compress_chunk **Bugfixes** * timescale#6541 Inefficient join plans on compressed hypertables. * timescale#6491 Enable now() plantime constification with BETWEEN * timescale#6494 Fix create_hypertable referenced by fk succeeds * timescale#6498 Suboptimal query plans when using time_bucket with query parameters * timescale#6507 time_bucket_gapfill with timezones doesn't handle daylight savings * timescale#6509 Make extension state available through function * timescale#6512 Log extension state changes * timescale#6522 Disallow triggers on CAggs * timescale#6523 Reduce locking level on compressed chunk index during segmentwise recompression * timescale#6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * timescale#6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * timescale#6575 Fix compressed chunk not found during upserts * timescale#6592 Fix recompression policy ignoring partially compressed chunks * timescale#6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

@coney21

This release contains performance improvements and bug fixes since the 2.13.1 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Ability to change compression settings on existing compressed hypertables at any time. New compression settings take effect on any new chunks that are compressed after the change. * Reduced locking requirements during chunk recompression * Limiting tuple decompression during DML operations to avoid decompressing a lot of tuples and causing storage issues (100k limit, configurable) * Helper functions for determining compression settings **For this release only**, you will need to restart the database before running `ALTER EXTENSION` **Multi-node support removal announcement** Following the deprecation announcement for Multi-node in TimescaleDB 2.13, Multi-node is no longer supported starting with TimescaleDB 2.14. TimescaleDB 2.13 is the last version that includes multi-node support. Learn more about it [here](docs/MultiNodeDeprecation.md). If you want to migrate from multi-node TimescaleDB to single-node TimescaleDB, read the [migration documentation](https://docs.timescale.com/migrate/latest/multi-node-to-timescale-service/). **Deprecation notice: recompress_chunk procedure** TimescaleDB 2.14 is the last version that will include the recompress_chunk procedure. Its functionality will be replaced by the compress_chunk function, which, starting on TimescaleDB 2.14, works on both uncompressed and partially compressed chunks. The compress_chunk function should be used going forward to fully compress all types of chunks or even recompress old fully compressed chunks using new compression settings (through the newly introduced recompress optional parameter). **Features** * #6325 Add plan-time chunk exclusion for real-time CAggs * #6360 Remove support for creating Continuous Aggregates with old format * #6386 Add functions for determining compression defaults * #6410 Remove multinode public API * #6440 Allow SQLValueFunction pushdown into compressed scan * #6463 Support approximate hypertable size * #6513 Make compression settings per chunk * #6529 Remove reindex_relation from recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6545 Remove restrictions for changing compression settings * #6566 Limit tuple decompression during DML operations * #6579 Change compress_chunk and decompress_chunk to idempotent version by default * #6608 Add LWLock for OSM usage in loader * #6609 Deprecate recompress_chunk * #6609 Add optional recompress argument to compress_chunk **Bugfixes** * #6541 Inefficient join plans on compressed hypertables. * #6491 Enable now() plantime constification with BETWEEN * #6494 Fix create_hypertable referenced by fk succeeds * #6498 Suboptimal query plans when using time_bucket with query parameters * #6507 time_bucket_gapfill with timezones doesn't handle daylight savings * #6509 Make extension state available through function * #6512 Log extension state changes * #6522 Disallow triggers on CAggs * #6523 Reduce locking level on compressed chunk index during segmentwise recompression * #6531 Fix if_not_exists behavior for CAgg policy with NULL offsets * #6571 Fix pathtarget adjustment for MergeAppend paths in aggregation pushdown code * #6575 Fix compressed chunk not found during upserts * #6592 Fix recompression policy ignoring partially compressed chunks * #6610 Ensure qsort comparison function is transitive **Thanks** * @coney21 and @GStechschulte for reporting the problem with inefficient join plans on compressed hypertables. * @HollowMan6 for reporting triggers not working on materialized views of CAggs * @jbx1 for reporting suboptimal query plans when using time_bucket with query parameters * @JerkoNikolic for reporting the issue with gapfill and DST * @pdipesh02 for working on removing the old Continuous Aggregate format * @raymalt and @martinhale for reporting very slow query plans on realtime CAggs queries

github-actions bot assigned svenklemm Jan 11, 2024

github-actions bot requested review from mkindahl and nikkhils January 11, 2024 09:23

svenklemm requested a review from antekresic January 11, 2024 09:27

antekresic reviewed Jan 11, 2024

View reviewed changes

tsl/test/expected/compression_ddl.out Show resolved Hide resolved

tsl/test/expected/compression_hypertable.out Outdated Show resolved Hide resolved

svenklemm force-pushed the per_chunk_settings branch from 746dffa to ea9e73e Compare January 11, 2024 12:31

akuzm reviewed Jan 11, 2024

View reviewed changes

tsl/src/compression/compression_storage.c Outdated Show resolved Hide resolved

akuzm reviewed Jan 11, 2024

View reviewed changes

tsl/test/expected/compression.out Show resolved Hide resolved

akuzm reviewed Jan 11, 2024

View reviewed changes

tsl/src/compression/compression_storage.c Outdated Show resolved Hide resolved

svenklemm force-pushed the per_chunk_settings branch 2 times, most recently from 446b598 to fe97f2a Compare January 12, 2024 12:50

svenklemm force-pushed the per_chunk_settings branch 2 times, most recently from b9d9daf to f5ccbc1 Compare January 13, 2024 15:04

antekresic self-requested a review January 15, 2024 11:39

antekresic approved these changes Jan 15, 2024

View reviewed changes

svenklemm force-pushed the per_chunk_settings branch 7 times, most recently from 5abb7c8 to c3dadea Compare January 15, 2024 21:24

erimatnor reviewed Jan 16, 2024

View reviewed changes

akuzm reviewed Jan 16, 2024

View reviewed changes

akuzm approved these changes Jan 16, 2024

View reviewed changes

svenklemm force-pushed the per_chunk_settings branch 3 times, most recently from 88c77ec to 0d3bd47 Compare January 17, 2024 11:30

svenklemm force-pushed the per_chunk_settings branch 2 times, most recently from 3427463 to a4d3a82 Compare January 17, 2024 11:37

svenklemm enabled auto-merge (rebase) January 17, 2024 11:52

svenklemm merged commit f57d584 into timescale:main Jan 17, 2024
43 checks passed

konskov mentioned this pull request Jan 23, 2024

Fix OSM build failure #6561

Merged

antekresic mentioned this pull request Feb 7, 2024

Release 2.14.0 #6613

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change compression settings to be per chunk #6513

Change compression settings to be per chunk #6513

svenklemm commented Jan 11, 2024

github-actions bot commented Jan 11, 2024

antekresic left a comment

codecov bot commented Jan 12, 2024 •

edited

antekresic left a comment

antekresic Jan 15, 2024

svenklemm Jan 17, 2024

antekresic Jan 15, 2024

erimatnor left a comment

erimatnor Jan 16, 2024

svenklemm Jan 17, 2024

erimatnor Jan 16, 2024

erimatnor Jan 16, 2024

erimatnor Jan 16, 2024

svenklemm Jan 17, 2024

akuzm Jan 16, 2024

svenklemm Jan 17, 2024

	TupleDesc tupdesc = rel->rd_att;
	TupleDesc tupdesc = RelationGetDescr(rel);

Change compression settings to be per chunk #6513

Change compression settings to be per chunk #6513

Conversation

svenklemm commented Jan 11, 2024

github-actions bot commented Jan 11, 2024

antekresic left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 12, 2024 • edited

Codecov Report

antekresic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 12, 2024 •

edited