Fix scankey where consttype is different from var #7055

nikkhils · 2024-06-21T11:36:18Z

When we decide to use an indexscan on the segment-by column for any query for decompression, then it's possible that the RHS constant type is not the same as the variable on the LHS of the comparison.

A typical example is "int8 = int4" comparison. While this works ok on 64 bit instances, it can crash on i386 ones. The issue is in the scankey that we build for segment-by columns. Even though we specify the valid "opcode" for cases where the arguments types don't match we also need to specify the "sk_subtype" appropriately.

Fixes #7039

svenklemm

Looks like you forgot to add the actual code change

nikkhils · 2024-06-21T12:52:39Z

Looks like you forgot to add the actual code change

Yup. I wanted to verify that the i386 test fails with this test addition. It does! 💯

erimatnor

I assume you did this to verify that the test reproduces the crash and that you will add the fix in another commit later?

nikkhils · 2024-06-21T12:54:06Z

I assume you did this to verify that the test reproduces the crash and that you will add the fix in another commit later?

heh. Both of you did the review so fast!

Actual fix added now. Please check @erimatnor @svenklemm

svenklemm · 2024-06-21T13:17:41Z

Does this not affect batch decompression in a similar way? @akuzm

erimatnor · 2024-06-21T13:56:38Z

tsl/src/compression/compression.c

+				if (typoid != castNode(Const, filter->value)->consttype)
+					subtype = castNode(Const, filter->value)->consttype;


Can't we just always include the subtype, even if it is the same?

+1 ... if you pass the consttype to the ScanKeyEntryInitialize you can get rid of this branch and also the variables you've created.

@erimatnor @fabriziomello I tried that. But it doesn't work in some cases:

ERROR: missing support function 1(1007,1007) in opfamily 1976

akuzm · 2024-06-21T18:16:37Z

Does this not affect batch decompression in a similar way? @akuzm

Which part exactly? If you're thinking about cross-type operators in vectorized filters, these should work fine.

When we decide to use an indexscan on the segment-by column for any query for decompression, then it's possible that the RHS constant type is not the same as the variable on the LHS of the comparison. A typical example is "int8 = int4" comparison. While this works ok on 64 bit instances, it can crash on i386 ones. The issue is in the scankey that we build for segment-by columns. Even though we specify the valid "opcode" for cases where the arguments types don't match we also need to specify the "sk_subtype" appropriately. Fixes timescale#7039

codecov · 2024-06-24T08:59:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.77%. Comparing base (59f50f2) to head (cea495e).
Report is 221 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7055      +/-   ##
==========================================
+ Coverage   80.06%   81.77%   +1.71%     
==========================================
  Files         190      200      +10     
  Lines       37181    37243      +62     
  Branches     9450     9705     +255     
==========================================
+ Hits        29770    30457     +687     
+ Misses       2997     2883     -114     
+ Partials     4414     3903     -511

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

timescale-automation · 2024-06-24T09:10:20Z

Automated backport to 2.15.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.15.x
You are currently cherry-picking commit 271053692.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .unreleased/fix_7055
	modified:   tsl/test/expected/compression.out
	modified:   tsl/test/sql/compression.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   tsl/src/compression/compression.c

Job log

This release contains performance improvements and bug fixes since the 2.15.3 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#6880: Add support for the array operators used for compressed DML batch filtering. * timescale#6895: Improve the compressed DML expression pushdown. * timescale#6897: Add support for replica identity on compressed hypertables. * timescale#6918: Remove support for PG13. * timescale#6920: Rework compression activity wal markers. * timescale#6989: Add support for foreign keys when converting plain tables to hypertables. * timescale#7020: Add support for the chunk column statistics tracking. * timescale#7048: Add an index scan for INSERT DML decompression. * timescale#7075: Reduce decompression on the compressed INSERT. * timescale#7101: Reduce decompressions for the compressed UPDATE/DELETE. * timescale#7108 Reduce decompressions for INSERTs with UNIQUE constraints **Bugfixes** * timescale#7018: Fix `search_path` quoting in the compression defaults function. * timescale#7046: Prevent locking for compressed tuples. * timescale#7055: Fix the `scankey` for `segment by` columns, where the type `constant` is different to `variable`. * timescale#7064: Fix the bug in the default `order by` calculation in compression. * timescale#7069: Fix the index column name usage. * timescale#7074: Fix the bug in the default `segment by` calculation in compression. **Thanks**

This release contains performance improvements and bug fixes since the 2.15.3 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#6880: Add support for the array operators used for compressed DML batch filtering. * timescale#6895: Improve the compressed DML expression pushdown. * timescale#6897: Add support for replica identity on compressed hypertables. * timescale#6918: Remove support for PG13. * timescale#6920: Rework compression activity wal markers. * timescale#6989: Add support for foreign keys when converting plain tables to hypertables. * timescale#7020: Add support for the chunk column statistics tracking. * timescale#7048: Add an index scan for INSERT DML decompression. * timescale#7075: Reduce decompression on the compressed INSERT. * timescale#7101: Reduce decompressions for the compressed UPDATE/DELETE. * timescale#7108 Reduce decompressions for INSERTs with UNIQUE constraints * timescale#7116 Use DELETE instead of TRUNCATE after compression * timescale#7134 Refactor foreign key handling for compressed hypertables **Bugfixes** * timescale#7018: Fix `search_path` quoting in the compression defaults function. * timescale#7046: Prevent locking for compressed tuples. * timescale#7055: Fix the `scankey` for `segment by` columns, where the type `constant` is different to `variable`. * timescale#7064: Fix the bug in the default `order by` calculation in compression. * timescale#7069: Fix the index column name usage. * timescale#7074: Fix the bug in the default `segment by` calculation in compression. **Thanks**

@jledentu

This release contains performance improvements and bug fixes since the 2.15.3 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#6880: Add support for the array operators used for compressed DML batch filtering. * timescale#6895: Improve the compressed DML expression pushdown. * timescale#6897: Add support for replica identity on compressed hypertables. * timescale#6918: Remove support for PG13. * timescale#6920: Rework compression activity wal markers. * timescale#6989: Add support for foreign keys when converting plain tables to hypertables. * timescale#7020: Add support for the chunk column statistics tracking. * timescale#7048: Add an index scan for INSERT DML decompression. * timescale#7075: Reduce decompression on the compressed INSERT. * timescale#7101: Reduce decompressions for the compressed UPDATE/DELETE. * timescale#7108 Reduce decompressions for INSERTs with UNIQUE constraints * timescale#7116 Use DELETE instead of TRUNCATE after compression * timescale#7134 Refactor foreign key handling for compressed hypertables * timescale#7161 Fix `mergejoin input data is out of order` **Bugfixes** * timescale#6987 Fix REASSIGN OWNED BY for background jobs * timescale#7018: Fix `search_path` quoting in the compression defaults function. * timescale#7046: Prevent locking for compressed tuples. * timescale#7055: Fix the `scankey` for `segment by` columns, where the type `constant` is different to `variable`. * timescale#7064: Fix the bug in the default `order by` calculation in compression. * timescale#7069: Fix the index column name usage. * timescale#7074: Fix the bug in the default `segment by` calculation in compression. **Thanks** * @jledentu For reporting a problem with mergejoin input order

@jledentu

This release contains significant performance improvements when working with compressed data, extended join support in continuous aggregates, and the ability to define foreign keys from regular tables towards hypertables. We recommend that you upgrade at the next available opportunity. In TimescaleDB v2.16.0 we: * Introduce multiple performance focused optimizations for data manipulation operations (DML) over compressed chunks. Improved upsert performance by more than 100x in some cases and more than 1000x in some update/delete scenarios. * Add the ability to define chunk skipping indexes on non-partitioning columns of compressed hypertables TimescaleDB v2.16.0 extends chunk exclusion to use those skipping (sparse) indexes when queries filter on the relevant columns, and prune chunks that do not include any relevant data for calculating the query response. * Offer new options for use cases that require foreign keys defined. You can now add foreign keys from regular tables towards hypertables. We have also removed some really annoying locks in the reverse direction that blocked access to referenced tables while compression was running. * Extend Continuous Aggregates to support more types of analytical queries. More types of joins are supported, additional equality operators on join clauses, and support for joins between multiple regular tables. **Highlighted features in this release** * Improved query performance through chunk exclusion on compressed hypertables. You can now define chunk skipping indexes on compressed chunks for any column with one of the following integer data types: `smallint`, `int`, `bigint`, `serial`, `bigserial`, `date`, `timestamp`, `timestamptz`. After you call `enable_chunk_skipping` on a column, TimescaleDB tracks the min and max values for that column. TimescaleDB uses that information to exclude chunks for queries that filter on that column, and would not find any data in those chunks. * Improved upsert performance on compressed hypertables. By using index scans to verify constraints during inserts on compressed chunks, TimescaleDB speeds up some ON CONFLICT clauses by more than 100x. * Improved performance of updates, deletes, and inserts on compressed hypertables. By filtering data while accessing the compressed data and before decompressing, TimescaleDB has improved performance for updates and deletes on all types of compressed chunks, as well as inserts into compressed chunks with unique constraints. By signaling constraint violations without decompressing, or decompressing only when matching records are found in the case of updates, deletes and upserts, TimescaleDB v2.16.0 speeds up those operations more than 1000x in some update/delete scenarios, and 10x for upserts. * You can add foreign keys from regular tables to hypertables, with support for all types of cascading options. This is useful for hypertables that partition using sequential IDs, and need to reference those IDs from other tables. * Lower locking requirements during compression for hypertables with foreign keys Advanced foreign key handling removes the need for locking referenced tables when new chunks are compressed. DML is no longer blocked on referenced tables while compression runs on a hypertable. * Improved support for queries on Continuous Aggregates `INNER/LEFT` and `LATERAL` joins are now supported. Plus, you can now join with multiple regular tables, and you can have more than one equality operator on join clauses. **PostgreSQL 13 support removal announcement** Following the deprecation announcement for PostgreSQL 13 in TimescaleDB v2.13, PostgreSQL 13 is no longer supported in TimescaleDB v2.16. The Currently supported PostgreSQL major versions are 14, 15 and 16. **Features** * #6880: Add support for the array operators used for compressed DML batch filtering. * #6895: Improve the compressed DML expression pushdown. * #6897: Add support for replica identity on compressed hypertables. * #6918: Remove support for PG13. * #6920: Rework compression activity wal markers. * #6989: Add support for foreign keys when converting plain tables to hypertables. * #7020: Add support for the chunk column statistics tracking. * #7048: Add an index scan for INSERT DML decompression. * #7075: Reduce decompression on the compressed INSERT. * #7101: Reduce decompressions for the compressed UPDATE/DELETE. * #7108 Reduce decompressions for INSERTs with UNIQUE constraints * #7116 Use DELETE instead of TRUNCATE after compression * #7134 Refactor foreign key handling for compressed hypertables * #7161 Fix `mergejoin input data is out of order` **Bugfixes** * #6987 Fix REASSIGN OWNED BY for background jobs * #7018: Fix `search_path` quoting in the compression defaults function. * #7046: Prevent locking for compressed tuples. * #7055: Fix the `scankey` for `segment by` columns, where the type `constant` is different to `variable`. * #7064: Fix the bug in the default `order by` calculation in compression. * #7069: Fix the index column name usage. * #7074: Fix the bug in the default `segment by` calculation in compression. **Thanks** * @jledentu For reporting a problem with mergejoin input order

nikkhils self-assigned this Jun 21, 2024

nikkhils force-pushed the decomp_del_i386 branch from 0c3ff0e to 2fb2250 Compare June 21, 2024 11:38

nikkhils requested review from erimatnor and svenklemm June 21, 2024 11:38

svenklemm reviewed Jun 21, 2024

View reviewed changes

erimatnor reviewed Jun 21, 2024

View reviewed changes

nikkhils force-pushed the decomp_del_i386 branch from 2fb2250 to 3bf2994 Compare June 21, 2024 12:53

nikkhils requested review from erimatnor and svenklemm June 21, 2024 12:54

svenklemm approved these changes Jun 21, 2024

View reviewed changes

erimatnor reviewed Jun 21, 2024

View reviewed changes

erimatnor approved these changes Jun 21, 2024

View reviewed changes

nikkhils force-pushed the decomp_del_i386 branch from 3bf2994 to c0e95a8 Compare June 24, 2024 07:55

nikkhils force-pushed the decomp_del_i386 branch from c0e95a8 to cea495e Compare June 24, 2024 08:49

nikkhils merged commit 2710536 into timescale:main Jun 24, 2024
38 checks passed

nikkhils deleted the decomp_del_i386 branch June 24, 2024 09:08

timescale-automation added the auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) label Jun 24, 2024

pallavisontakke added this to the TimescaleDB 2.16.0 milestone Jul 12, 2024

pallavisontakke mentioned this pull request Jul 18, 2024

Release 2.16.0 #7135

Closed

pallavisontakke mentioned this pull request Jul 25, 2024

Release 2.16.0 #7156

Closed

pallavisontakke mentioned this pull request Jul 31, 2024

Release 2.16.0 #7169

Closed

bayandin mentioned this pull request Aug 1, 2024

timescaledb 2.16.0 bayandin/homebrew-tap#173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix scankey where consttype is different from var #7055

Fix scankey where consttype is different from var #7055

nikkhils commented Jun 21, 2024

svenklemm left a comment

nikkhils commented Jun 21, 2024

erimatnor left a comment

nikkhils commented Jun 21, 2024

svenklemm commented Jun 21, 2024

erimatnor Jun 21, 2024

fabriziomello Jun 22, 2024

nikkhils Jun 24, 2024

akuzm commented Jun 21, 2024

codecov bot commented Jun 24, 2024

timescale-automation commented Jun 24, 2024

		if (typoid != castNode(Const, filter->value)->consttype)
		subtype = castNode(Const, filter->value)->consttype;

Fix scankey where consttype is different from var #7055

Fix scankey where consttype is different from var #7055

Conversation

nikkhils commented Jun 21, 2024

svenklemm left a comment

Choose a reason for hiding this comment

nikkhils commented Jun 21, 2024

erimatnor left a comment

Choose a reason for hiding this comment

nikkhils commented Jun 21, 2024

svenklemm commented Jun 21, 2024

erimatnor Jun 21, 2024

Choose a reason for hiding this comment

fabriziomello Jun 22, 2024

Choose a reason for hiding this comment

nikkhils Jun 24, 2024

Choose a reason for hiding this comment

akuzm commented Jun 21, 2024

codecov bot commented Jun 24, 2024

Codecov Report

timescale-automation commented Jun 24, 2024

Git status