Release 2.3.1 #3373

mkindahl · 2021-06-29T11:40:00Z

Commit message is

This maintenance release contains bugfixes since the 2.3.0 release. We
deem it moderate priority for upgrading.

**Bugfixes**
* #3279 Add some more randomness to chunk assignment
* #3288 Fix failed update with parallel workers
* #3300 Improve trigger handling on distributed hypertables
* #3304 Remove paths that reference parent relids for compressed chunks
* #3305 Fix pull_varnos miscomputation of relids set
* #3310 Generate downgrade script
* #3314 Fix heap buffer overflow in hypertable expansion
* #3317 Fix heap buffer overflow in remote connection cache.
* #3327 Make aggregate in caggs fully qualified
* #3336 Fix pg_init_privs objsubid handling
* #3345 Fix SkipScan distinct column identification
* #3355 Fix heap buffer overflow when renaming compressed hypertable columns.
* #3367 Improve DecompressChunk qual pushdown
* #3377 Fix bad use of repalloc

**Thanks**
* @db-adrian for reporting an issue when accessing cagg view through postgres_fdw
* @fncaldas and @pgwhalen for reporting an issue accessing caggs when public is not in search_path
* @fvannee, @mglonnro and @ebreijo for reporting an issue with the upgrade script
* @fvannee for reporting a performance regression with SkipScan

Commits for this PR are:

9a0caef3 Fix bad use of repalloc
3b1b6636 Update version
78349747 Export variables to docker-build
2e38c744 Always define 'format' target
49ec2c7c Update test files
20877dda Fix output file for compression_qualpushdown
c6b26a74 Improve DecompressChunk qual pushdown
b9da4879 Fix ABI breakage test in CI
63f6ae53 Fix memory overflow
3a669a45 Generate downgrade script
a7e79bd2 Stabilize timestamp_limits test output
d6bf30b9 Fix SkipScan distinct column identification
95154fc3 Add compatibility macro for convert_tuples_by_name
a4e8ecfc Move timestamp_limits and with_clause_parser test
66a7ebf8 Fix memory overflow
dacf9211 Fix Regression Linux i386 workflow
e882bcff Run sanitizer tests on PG12
b71bf163 Fix update tests to handle sequences
a0082cb1 Format CMake files
3003c848 Filter out chunk and hypertable ids from test output
c71f42e5 Improve trigger handling on distributed hypertables
3329c01d Harden pg_init_privs query
b82999eb Fix flaky pg_dump test
e171255f Fix failed update with parallel workers
a198ce5e Fix pg_init_privs objsubid handling
3084fa7e Constify parameters and return values of core APIs
9d634be3 Make aggregate in caggs fully qualified
bef7c086 Fix compile error
5f74a392 Fix pull_varnos miscomputation of relids set
98389656 Remove paths that reference parent relids for compressed chunks
ce5438f6 Fix memory overflow
67eacf8f Remove unused bucket_width argument
07407cdf Add some more randomness to chunk assignment
9b36f72c Add missing ORDER BY clause to dist_hypertable test
48b30eb6 Fix flaky pg_dump test
5b64d627 Fix broken links in TimescaleDB NOTICE
805a9f66 Add 2.3.0 to update test scripts
846b468f Fix issues in update smoke test
cedf2e14 Remove unused job_type from policies
fe8910e2 Use correct heading level in changelog

codecov · 2021-06-29T12:19:59Z

Codecov Report

❗ No coverage uploaded for pull request base (2.3.x@fe8910e). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 8bbcb5b differs from pull request most recent head 6d90bd7. Consider uploading reports for the commit 6d90bd7 to get more accurate results

@@           Coverage Diff            @@
##             2.3.x    #3373   +/-   ##
========================================
  Coverage         ?   90.47%           
========================================
  Files            ?      216           
  Lines            ?    36079           
  Branches         ?        0           
========================================
  Hits             ?    32644           
  Misses           ?     3435           
  Partials         ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe8910e...6d90bd7. Read the comment docs.

svenklemm

I dont think PG11 support should be removed in 2.3.x branch

When refactoring the job code the job_type was removed from our catalog but a few places got overlooked that still had a variable holding the job type.

Corrected a function name and replaced a variable with the explicit commands to use.

The update tests for PG11, PG12, and PG13 are updated to include release 2.3.0.

Link to Telemetry page explains more than disable it Co-authored-by: Mike Freedman <mfreed@cs.princeton.edu>

Stop background workers in test after calling timescaledb_post_restore since this function restarts background workers.

mkindahl · 2021-06-30T16:09:49Z

I dont think PG11 support should be removed in 2.3.x branch

PG11 removed from the branch. Needed to update some tests as well, but I'll merge those fixes with the associated changes.

Previously the assignment of data nodes to chunks had a bit of a thundering-herd problem for multiple hypertables without space partions: the data node assigned for the first chunk was always the same across hypertables. We fix this by adding the hypertable_id to the index into the datanode array. This de-synchronizes across hypertables but maintains consistency for any given hypertable. We could make this consistent for space partitioned tables as well but avoid doing so now to prevent partitions jumping nodes due to this change. This also effects tablespace selection in the same way.

continuous_agg_update_materialization() doesn't use this argument and we can't rely on a fixed bucket_width in the future anyway.

NunoFilipeSantos

Thank you Mats, this looks great.

svenklemm · 2021-07-05T14:43:34Z

CHANGELOG.md

+## 2.3.1 (2021-07-05)
+
+This maintenance release contains bugfixes since the 2.3.0 release. We
+deem it moderate priority for upgrading.


We usually do a summary of the areas covered by the bugfixes

For patch releases, I find that redundant since the bugs are listed just below, but I added a blurb about that to the changelog for this release.

erimatnor · 2021-07-05T14:46:55Z

sql/CMakeLists.txt

@@ -40,7 +40,8 @@ set(MOD_FILES
    updates/2.1.0--2.1.1.sql
    updates/2.1.1--2.2.0.sql
    updates/2.2.0--2.2.1.sql
-    updates/2.2.1--2.3.0.sql)
+    updates/2.2.1--2.3.0.sql
+    updates/2.3.0--2.3.1.sql)


Should I expect something like this also for the downgrade path?

Yes. The downgrade path need to be added after the release is tagged since the downgrade tests check out a tagged release, so it is not added here.

Hmm.... let me check this again....

Thanks for catching it. Had to update the downgrade list at this step as well.

erimatnor · 2021-07-05T14:50:06Z

sql/updates/2.3.0--2.3.1.sql

@@ -0,0 +1 @@
+DROP FUNCTION IF EXISTS _timescaledb_internal.refresh_continuous_aggregate;


This function is dropped because it moved to the experimental schema right? But I didn't see we are including the experimental schema in this release, so is this correct?

I guess dropping the function doesn't hurt, though, since functions are recreated during the upgrade if I am not wrong. Would still like to understand why this has changed in this release.

No, the function is dropped because it is re-defined. We do not introduce the experimental schema in this release since it causes issues with the update tests and we need to go over it separately.

erimatnor · 2021-07-05T14:51:06Z

version.config

@@ -1,4 +1,3 @@
 version = 2.3.1
 update_from_version = 2.3.0
 downgrade_to_version = 2.3.0
-


What is this commit for?

Seems to be remains from an update patch from master. Removed it.

This commit add functions and code to generate a downgrade script from the current version to the previous version. This requires execution from a Git repository since it retrieves the prolog and epilog for the "downgrade" file from the version given by `update_from_version` in the `version.config` file. The commit adds several CMake functions that simplifies the composition of script files, but these are not used to generate the update scripts. A potential improvement is to use the scripts to also generate the update scripts. This commit supports generating a downgrade script from the current version to the previous version. Other versions are handled using a variable containing file names of reverse update scripts and the source and target version is extracted from the file names, which is assumed to be of the form `<source-version>--<target-version>.sql`. In addition to adding support for generating downgrade scripts, the commit adds a downgrade test file that tests a release in a similar way to the update script and adds it as a workflow.

Fix heap buffer overflow in hypertable expansion.

Add the files that had been missing from the docker environment since the introduction of the downgrade scripts.

Allow pushdown of RelabelType expressions into the scan below the DecompressChunk node. When using varchar columns as segmentby columns constraints on those columns would not be pushed down because postgres would inject RelabelType nodes that where not accepted as valid expression for pushdown leading to bad performance because the filter could only be applied after decompression. Fixes timescale#3351

The output file `compression_qualpushdown-11.out` was not updated.

Output for `compression_qualpushdown-13.out` was not correct.

If `clang-format` is not found `CLANG_FORMAT` will be set to a false value, which will prevent the `format` target from being defined if `cmake-format` is not installed. This commit fixes that by always creating a `format` target since `clang-format` is always created even if `CLANG_FORMAT` is false. We do not redefine `CLANG_FORMAT` to a non-false value since it is expected to contain a path to an executable and if set to a true value that is not a path it could be used in a way that leads to strange errors.

This commit export the `GENERATE_DOWNGRADE_SCRIPT` and `PG_VERSION` variables from `test_downgrade_to_tag.sh` to `docker-build.sh`.

In contrast to `realloc`, the PostgreSQL function `repalloc` does not accept a NULL pointer and fall back on `palloc`. For that reason it is necessary to check if the pointer is NULL and either use `palloc` or `repalloc`.

During an update, it is not possible to run the downgrade scripts until the release has been tagged, but the update scripts can be run. This means that we need to split the previous version into two different fields: one for running the update tests and one for running the downgrade tests.

@fncaldas

This maintenance release contains bugfixes since the 2.3.0 release. We deem it moderate priority for upgrading. The release introduces the possibility of generating downgrade scripts, improves the trigger handling for distributed hypertables, adds some more randomness to chunk assignment to avoid thundering herd issues in chunk assignment, and fixes some issues in update handling as well as some other bugs. **Bugfixes** * timescale#3279 Add some more randomness to chunk assignment * timescale#3288 Fix failed update with parallel workers * timescale#3300 Improve trigger handling on distributed hypertables * timescale#3304 Remove paths that reference parent relids for compressed chunks * timescale#3305 Fix pull_varnos miscomputation of relids set * timescale#3310 Generate downgrade script * timescale#3314 Fix heap buffer overflow in hypertable expansion * timescale#3317 Fix heap buffer overflow in remote connection cache. * timescale#3327 Make aggregate in caggs fully qualified * timescale#3327 Make aggregates in caggs fully qualified * timescale#3336 Fix pg_init_privs objsubid handling * timescale#3345 Fix SkipScan distinct column identification * timescale#3355 Fix heap buffer overflow when renaming compressed hypertable columns. * timescale#3367 Improve DecompressChunk qual pushdown * timescale#3377 Fix bad use of repalloc **Thanks** * @db-adrian for reporting an issue when accessing cagg view through postgres_fdw * @fncaldas and @pgwhalen for reporting an issue accessing caggs when public is not in search_path * @fvannee, @mglonnro and @ebreijo for reporting an issue with the upgrade script * @fvannee for reporting a performance regression with SkipScan

mkindahl requested a review from a team as a code owner June 29, 2021 11:40

mkindahl requested review from afiskon, svenklemm and gayyappan and removed request for a team June 29, 2021 11:40

mkindahl force-pushed the release-2.3.1 branch from 4ad1cb7 to 8f2e19c Compare June 29, 2021 12:05

mkindahl force-pushed the release-2.3.1 branch 2 times, most recently from 0074235 to 2ed9d32 Compare June 29, 2021 12:27

svenklemm requested changes Jun 29, 2021

View reviewed changes

mkindahl force-pushed the release-2.3.1 branch 2 times, most recently from f9ea18a to 5e55155 Compare June 29, 2021 17:03

svenklemm and others added 5 commits June 29, 2021 19:33

Remove unused job_type from policies

cedf2e1

When refactoring the job code the job_type was removed from our catalog but a few places got overlooked that still had a variable holding the job type.

Fix issues in update smoke test

846b468

Corrected a function name and replaced a variable with the explicit commands to use.

Add 2.3.0 to update test scripts

805a9f6

The update tests for PG11, PG12, and PG13 are updated to include release 2.3.0.

Fix broken links in TimescaleDB NOTICE

5b64d62

Link to Telemetry page explains more than disable it Co-authored-by: Mike Freedman <mfreed@cs.princeton.edu>

Fix flaky pg_dump test

48b30eb

Stop background workers in test after calling timescaledb_post_restore since this function restarts background workers.

mkindahl force-pushed the release-2.3.1 branch 9 times, most recently from 8bfe994 to a5fb316 Compare June 30, 2021 14:47

svenklemm and others added 3 commits June 30, 2021 18:11

Add missing ORDER BY clause to dist_hypertable test

9b36f72

Remove unused bucket_width argument

67eacf8

continuous_agg_update_materialization() doesn't use this argument and we can't rely on a fixed bucket_width in the future anyway.

mkindahl requested a review from svenklemm July 1, 2021 13:38

mkindahl force-pushed the release-2.3.1 branch 3 times, most recently from 627dea3 to 799f5a6 Compare July 2, 2021 08:33

mkindahl requested a review from NunoFilipeSantos July 2, 2021 08:57

afiskon approved these changes Jul 2, 2021

View reviewed changes

mkindahl self-assigned this Jul 2, 2021

mkindahl force-pushed the release-2.3.1 branch from 799f5a6 to 9d7dab3 Compare July 2, 2021 10:41

NunoFilipeSantos approved these changes Jul 2, 2021

View reviewed changes

svenklemm reviewed Jul 5, 2021

View reviewed changes

erimatnor reviewed Jul 5, 2021

View reviewed changes

mkindahl force-pushed the release-2.3.1 branch 2 times, most recently from f09fca3 to 2bbfab3 Compare July 5, 2021 15:23

svenklemm approved these changes Jul 5, 2021

View reviewed changes

mkindahl and others added 12 commits July 5, 2021 22:38

Fix memory overflow

643bb18

Fix heap buffer overflow in hypertable expansion.

Fix ABI breakage test in CI

b76f54a

Add the files that had been missing from the docker environment since the introduction of the downgrade scripts.

Fix output file for compression_qualpushdown

211d572

The output file `compression_qualpushdown-11.out` was not updated.

Update test files

90ec629

Output for `compression_qualpushdown-13.out` was not correct.

Export variables to docker-build

9cf1915

This commit export the `GENERATE_DOWNGRADE_SCRIPT` and `PG_VERSION` variables from `test_downgrade_to_tag.sh` to `docker-build.sh`.

Fix bad use of repalloc

c71b17a

In contrast to `realloc`, the PostgreSQL function `repalloc` does not accept a NULL pointer and fall back on `palloc`. For that reason it is necessary to check if the pointer is NULL and either use `palloc` or `repalloc`.

Add downgrade script

7d8e189

mkindahl force-pushed the release-2.3.1 branch from 8bbcb5b to 6d90bd7 Compare July 5, 2021 20:39

mkindahl requested a review from erimatnor July 5, 2021 20:41

mkindahl merged commit a42284f into timescale:2.3.x Jul 6, 2021

mkindahl deleted the release-2.3.1 branch July 6, 2021 06:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 2.3.1 #3373

Release 2.3.1 #3373

mkindahl commented Jun 29, 2021 •

edited

codecov bot commented Jun 29, 2021 •

edited

svenklemm left a comment

mkindahl commented Jun 30, 2021

NunoFilipeSantos left a comment

svenklemm Jul 5, 2021

mkindahl Jul 5, 2021

erimatnor Jul 5, 2021

mkindahl Jul 5, 2021

mkindahl Jul 5, 2021

mkindahl Jul 5, 2021

erimatnor Jul 5, 2021 •

edited

mkindahl Jul 5, 2021

erimatnor Jul 5, 2021

mkindahl Jul 5, 2021

		@@ -0,0 +1 @@
		DROP FUNCTION IF EXISTS _timescaledb_internal.refresh_continuous_aggregate;

Release 2.3.1 #3373

Release 2.3.1 #3373

Conversation

mkindahl commented Jun 29, 2021 • edited

codecov bot commented Jun 29, 2021 • edited

Codecov Report

svenklemm left a comment

Choose a reason for hiding this comment

mkindahl commented Jun 30, 2021

NunoFilipeSantos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor Jul 5, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkindahl commented Jun 29, 2021 •

edited

codecov bot commented Jun 29, 2021 •

edited

erimatnor Jul 5, 2021 •

edited