Support for CAgg with time_bucket origin/offset parameter #6382

jnidzwetzki · 2023-12-06T16:48:36Z

So far, we allowed only CAggs without origin or offset parameters in the
time_bucket definition. This commit adds support for the remaining
time_bucket variants.

Fixes #2265, Fixes #5453, Fixes #5828

Needed changes

Allow additional time_bucket variants
Enhance time_bucket query parsing in CAgg code to capture offset/origin properly
Align refresh window
Call proper time_bucket functions (offset and origin are handled by different implementations)
Restrict use of broken time_bucket queries (i.e., variable bucket with offset)

Todo

Tasks

Give feedback

Handle timezone information for non variable-sized buckets

Handle timezone information for non variable-sized buckets
Options
Successfully updated the issue's project

There was an error updating the issue's project
Test real-time feature

Test real-time feature
Options
Successfully updated the issue's project

There was an error updating the issue's project
Move origin/offset into time_bucket catalog table

Move origin/offset into time_bucket catalog table
Options
Successfully updated the issue's project

There was an error updating the issue's project
Fix problems with refresh and invalidations

Fix problems with refresh and invalidations
Options
Successfully updated the issue's project

There was an error updating the issue's project
Test integer based buckets

Test integer based buckets
Options
Successfully updated the issue's project

There was an error updating the issue's project
Test creation of Cagg with different bucket definitions (named, unnamed)

Test creation of Cagg with different bucket definitions (named, unnamed)
Options
Successfully updated the issue's project

There was an error updating the issue's project
Test HCaggs

Test HCaggs
Options
Successfully updated the issue's project

There was an error updating the issue's project
Block HCagg with incompatible origin/offset values

Block HCagg with incompatible origin/offset values
Options
Successfully updated the issue's project

There was an error updating the issue's project
Block fixed offsets with variable buckets

Block fixed offsets with variable buckets
Options
Successfully updated the issue's project

There was an error updating the issue's project
Improve test coverage

Improve test coverage
Options
Successfully updated the issue's project

There was an error updating the issue's project
Options

codecov · 2023-12-14T11:27:03Z

Codecov Report

Attention: Patch coverage is 81.87500% with 29 lines in your changes are missing coverage. Please review.

Project coverage is 80.94%. Comparing base (59f50f2) to head (2a5813b).
Report is 93 commits behind head on main.

Files	Patch %	Lines
tsl/src/continuous_aggs/refresh.c	71.42%	4 Missing and 12 partials ⚠️
tsl/src/continuous_aggs/common.c	83.82%	2 Missing and 9 partials ⚠️
src/time_bucket.c	94.73%	0 Missing and 1 partial ⚠️
tsl/src/continuous_aggs/create.c	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6382      +/-   ##
==========================================
+ Coverage   80.06%   80.94%   +0.87%     
==========================================
  Files         190      194       +4     
  Lines       37181    36853     -328     
  Branches     9450     9647     +197     
==========================================
+ Hits        29770    29830      +60     
- Misses       2997     3174     +177     
+ Partials     4414     3849     -565

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mkindahl

Looks good. I suggested to add a few comments.

mkindahl · 2024-04-03T08:02:20Z

tsl/test/expected/cagg_ddl-16.out

I don't follow. The description says that you "[Add] Support for CAgg with time_bucket origin/offset parameter", but this test shows that it does not support it. What am I missing?

(see discussion below)

tsl/test/sql/cagg_ddl.sql.in

mkindahl · 2024-04-03T08:12:56Z

tsl/src/continuous_aggs/common.c

+		/* Timezone as text */
+		case TEXTOID:
+			tz_name = TextDatumGetCString(arg->constvalue);
+			if (!ts_is_valid_timezone_name(tz_name))


Not sure why we are not using DecodeTimezoneName or DecodeTimezoneNameToTz? I realize this function already existed, so this is just an observation. Nothing to do for this PR.

tsl/src/continuous_aggs/common.c

mkindahl · 2024-04-04T06:59:32Z

Hello! Would this PR be able to close issue #5806 ? I don't see them linked. Thank you!

@guiofranca I suspect not. This issue is about being able to use offset and origin parameters in caggs, but the referenced issue is about having inclusive and exclusive ranges for the buckets.

src/time_bucket.h

tsl/src/continuous_aggs/common.c

tsl/test/sql/cagg_query.sql

fabriziomello

Left some minor comments but other than that LGTM!

guiofranca · 2024-04-04T14:52:56Z

Hello! Would this PR be able to close issue #5806 ? I don't see them linked. Thank you!

@guiofranca I suspect not. This issue is about being able to use offset and origin parameters in caggs, but the referenced issue is about having inclusive and exclusive ranges for the buckets.

Hey, thanks for the reply @mkindahl ! I understand that is being added support for origin and offset for the time_bucket, but shifting the interval will not be suported as @jnidzwetzki said earlier.

So the following time bucket for the CAgg would still not be supported because of the shifting done:

time_bucket('15 min', instant, '1 second'::interval) - '1 second'::interval

The thing I'm looking forward to test, when released, is the following statement:

time_bucket('15 min', instant, '1 microsecond'::interval) as bucket

This would not change how timestamp is displayed to the second, but would change how I query the data, requiring me to add that offset to my query range.

Anyway, grouping the data with the correct offset is what I'm mainlingly looking for, the shifting part is manageable outside the CAgg.

So far, we allowed only CAggs without origin or offset parameters in the time_bucket definition. This commit adds support for the remaining time_bucket variants. Fixes timescale#2265, Fixes timescale#5453, Fixes timescale#5828

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

ffernest · 2024-06-06T22:51:32Z

Hi, I tried v2.15.1 today with the below command. It seems that the offset and origin parameters are still not working. Is there something I am missing? Thanks!

dbdev=# CREATE MATERIALIZED VIEW vtstable WITH (timescaledb.continuous, timescaledb.materialized_only = FALSE) AS
SELECT time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00') AS time,
max(volume) AS maxcount, sum(volume) AS totalcount FROM tstable GROUP BY time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00');
ERROR:  cannot create continuous aggregate with variable-width bucket using offset or origin.

fabriziomello · 2024-06-06T23:38:00Z

Hi, I tried v2.15.1 today with the below command. It seems that the offset and origin parameters are still not working. Is there something I am missing? Thanks!


dbdev=# CREATE MATERIALIZED VIEW vtstable WITH (timescaledb.continuous, timescaledb.materialized_only = FALSE) AS

SELECT time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00') AS time,

max(volume) AS maxcount, sum(volume) AS totalcount FROM tstable GROUP BY time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00');

ERROR:  cannot create continuous aggregate with variable-width bucket using offset or origin.

This was a restriction that will be removed in the upcoming 2.15.2. I'm planning to tag the release and build packages and docker images tomorrow.

ffernest · 2024-06-07T13:29:55Z

Hi, I tried v2.15.1 today with the below command. It seems that the offset and origin parameters are still not working. Is there something I am missing? Thanks!
dbdev=# CREATE MATERIALIZED VIEW vtstable WITH (timescaledb.continuous, timescaledb.materialized_only = FALSE) AS

SELECT time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00') AS time,

max(volume) AS maxcount, sum(volume) AS totalcount FROM tstable GROUP BY time_bucket(INTERVAL '4 HOURS', time, 'America/New_York', TIMESTAMPTZ '2000-01-01 22:00:00+00');

ERROR:  cannot create continuous aggregate with variable-width bucket using offset or origin.
This was a restriction that will be removed in the upcoming 2.15.2. I'm planning to tag the release and build packages and docker images tomorrow.

Thanks for your prompt reply. I can't wait to try this feature.

github-actions bot assigned jnidzwetzki Dec 6, 2023

jnidzwetzki force-pushed the cagg_bucket branch 11 times, most recently from 931f862 to d4b4e3b Compare December 13, 2023 13:55

jnidzwetzki force-pushed the cagg_bucket branch 2 times, most recently from 36c7b22 to 4c80713 Compare December 14, 2023 07:48

jnidzwetzki changed the title ~~CAgg experiments~~ Support for CAgg with time_bucket origin/offset parameter Jan 24, 2024

jnidzwetzki force-pushed the cagg_bucket branch 5 times, most recently from 86547ec to 592f473 Compare February 1, 2024 14:45

jnidzwetzki mentioned this pull request Feb 23, 2024

Use NULL in CAgg bucket function catalog table #6682

Merged

fabriziomello added time_bucket continuous_aggregate labels Mar 6, 2024

jnidzwetzki force-pushed the cagg_bucket branch 6 times, most recently from b1c996b to d440a77 Compare March 14, 2024 13:25

jnidzwetzki marked this pull request as ready for review March 30, 2024 08:37

jnidzwetzki force-pushed the cagg_bucket branch from 4e4b260 to f6b0f33 Compare March 30, 2024 08:39

jnidzwetzki mentioned this pull request Apr 2, 2024

Prevent usage of time_bucket_ng in CAgg definition #6798

Merged

jnidzwetzki force-pushed the cagg_bucket branch 3 times, most recently from 1b1759a to ac0a1bd Compare April 2, 2024 08:54

mkindahl approved these changes Apr 3, 2024

View reviewed changes

jnidzwetzki force-pushed the cagg_bucket branch from ac0a1bd to 76628d2 Compare April 4, 2024 06:47

jnidzwetzki added this to the TimescaleDB 2.15.0 milestone Apr 4, 2024

fabriziomello reviewed Apr 4, 2024

View reviewed changes

src/time_bucket.h Outdated Show resolved Hide resolved

tsl/src/continuous_aggs/common.c Outdated Show resolved Hide resolved

tsl/test/sql/cagg_query.sql Show resolved Hide resolved

tsl/test/sql/cagg_query.sql Outdated Show resolved Hide resolved

fabriziomello approved these changes Apr 4, 2024

View reviewed changes

jnidzwetzki force-pushed the cagg_bucket branch from 76628d2 to 232b869 Compare April 5, 2024 07:22

Support for CAgg with origin/offset parameter

2a5813b

So far, we allowed only CAggs without origin or offset parameters in the time_bucket definition. This commit adds support for the remaining time_bucket variants. Fixes timescale#2265, Fixes timescale#5453, Fixes timescale#5828

jnidzwetzki force-pushed the cagg_bucket branch from 232b869 to 2a5813b Compare April 5, 2024 07:26

jnidzwetzki merged commit 8d9b062 into timescale:main Apr 5, 2024
43 checks passed

jnidzwetzki deleted the cagg_bucket branch April 5, 2024 08:31

fabriziomello mentioned this pull request Apr 30, 2024

Release notes header 2.15.0 #6874

Merged

fabriziomello mentioned this pull request May 6, 2024

Release 2.15.0 #6887

Merged

fabriziomello mentioned this pull request May 6, 2024

Release 2.15.0: Release branch #6889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for CAgg with time_bucket origin/offset parameter #6382

Support for CAgg with time_bucket origin/offset parameter #6382

jnidzwetzki commented Dec 6, 2023 •

edited

Loading

Tasks

codecov bot commented Dec 14, 2023 •

edited

Loading

mkindahl left a comment

mkindahl Apr 3, 2024

jnidzwetzki Apr 4, 2024

mkindahl Apr 3, 2024

mkindahl commented Apr 4, 2024 •

edited

Loading

fabriziomello left a comment

guiofranca commented Apr 4, 2024

ffernest commented Jun 6, 2024

fabriziomello commented Jun 6, 2024

ffernest commented Jun 7, 2024

Support for CAgg with time_bucket origin/offset parameter #6382

Support for CAgg with time_bucket origin/offset parameter #6382

Conversation

jnidzwetzki commented Dec 6, 2023 • edited Loading

Needed changes

Todo

Tasks

codecov bot commented Dec 14, 2023 • edited Loading

Codecov Report

mkindahl left a comment

Choose a reason for hiding this comment

mkindahl Apr 3, 2024

Choose a reason for hiding this comment

jnidzwetzki Apr 4, 2024

Choose a reason for hiding this comment

mkindahl Apr 3, 2024

Choose a reason for hiding this comment

mkindahl commented Apr 4, 2024 • edited Loading

fabriziomello left a comment

Choose a reason for hiding this comment

guiofranca commented Apr 4, 2024

ffernest commented Jun 6, 2024

fabriziomello commented Jun 6, 2024

ffernest commented Jun 7, 2024

jnidzwetzki commented Dec 6, 2023 •

edited

Loading

codecov bot commented Dec 14, 2023 •

edited

Loading

mkindahl commented Apr 4, 2024 •

edited

Loading