Support UPDATE/DELETE on compressed hypertables. #5339

sb230132 · 2023-02-16T13:25:20Z

This patch does following:

Executor changes to parse qual ExprState to check if SEGMENTBY
column is specified in WHERE clause.
Based on step 1, we build scan keys.
Executor changes to do heapscan on compressed chunk based on
scan keys and move only those rows which match the WHERE clause
to staging are aka uncompressed chunk.
Mark affected chunk as partially compressed.
Perform regular UPDATE/DELETE operations on staging area aka
uncompressed chunk.
Since there is no Custom Scan (HypertableModify) node for
UPDATE/DELETE operations on PG versions < 14, we don't support this
feature on PG12 and PG13.

codecov · 2023-02-17T17:41:37Z

Codecov Report

Merging #5339 (07d48d7) into main (c2941a3) will decrease coverage by 0.35%.
The diff coverage is 95.32%.

@@            Coverage Diff             @@
##             main    #5339      +/-   ##
==========================================
- Coverage   90.75%   90.41%   -0.35%     
==========================================
  Files         229      229              
  Lines       53707    53871     +164     
==========================================
- Hits        48742    48705      -37     
- Misses       4965     5166     +201

Impacted Files	Coverage Δ
tsl/src/compression/compression.h	`0.00% <ø> (ø)`
tsl/src/init.c	`96.29% <ø> (ø)`
tsl/src/planner.c	`95.12% <ø> (-4.88%)`	⬇️
tsl/src/compression/compression.c	`95.93% <94.20%> (-0.25%)`	⬇️
src/nodes/hypertable_modify.c	`72.60% <100.00%> (+1.08%)`	⬆️

... and 18 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-02-20T04:22:12Z

@fabriziomello, @konskov: please review this pull request.

Powered by pull-review

svenklemm

Seems like you are completely missing the batch filtering. This needs way more tests especially isolation tests. Not sure why we need the modified DecompressChunkDML node since the tuples move into uncompressed chunk this should not be needed. Did you have a look at how this is handled in the INSERT path? Why did you choose a different approach here?

sb230132 · 2023-02-28T09:24:01Z

Seems like you are completely missing the batch filtering. This needs way more tests especially isolation tests. Not sure why we need the modified DecompressChunkDML node since the tuples move into uncompressed chunk this should not be needed. Did you have a look at how this is handled in the INSERT path? Why did you choose a different approach here?

Fixed as per your suggestion.

src/nodes/hypertable_modify.c

src/nodes/hypertable_modify.h

tsl/src/compression/compression.c

CHANGELOG.md

This patch does following: 1. Executor changes to parse qual ExprState to check if SEGMENTBY column is specified in WHERE clause. 2. Based on step 1, we build scan keys. 3. Executor changes to do heapscan on compressed chunk based on scan keys and move only those rows which match the WHERE clause to staging area aka uncompressed chunk. 4. Mark affected chunk as partially compressed. 5. Perform regular UPDATE/DELETE operations on staging area. 6. Since there is no Custom Scan (HypertableModify) node for UPDATE/DELETE operations on PG versions < 14, we don't support this feature on PG12 and PG13.

@kovetskiy

This release includes these noteworthy features: * compressed hypertable enhancements: * UPDATE/DELETE support * ON CONFLICT DO UPDATE * Join support for hierarchical Continougs Aggregates * performance improvements **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

Micromegass · 2023-05-16T14:49:41Z

@sb230132 Since this is a very important feature for me (and probably for many), is there any info when this feature will be released?

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * #5212 Allow pushdown of reference table joins * #5221 Improve Realtime Continuous Aggregate performance * #5252 Improve unique constraint support on compressed hypertables * #5339 Support UPDATE/DELETE on compressed hypertables * #5344 Enable JOINS for Hierarchical Continuous Aggregates * #5361 Add parallel support for partialize_agg() * #5417 Refactor and optimize distributed COPY * #5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * #5547 Skip Ordered Append when only 1 child node is present * #5510 Propagate vacuum/analyze to compressed chunks * #5584 Reduce decompression during constraint checking * #5530 Optimize compressed chunk resorting * #5639 Support sending telemetry event reports **Bugfixes** * #5396 Fix SEGMENTBY columns predicates to be pushed down * #5427 Handle user-defined FDW options properly * #5442 Decompression may have lost DEFAULT values * #5459 Fix issue creating dimensional constraints * #5570 Improve interpolate error message on datatype mismatch * #5573 Fix unique constraint on compressed tables * #5615 Add permission checks to run_job() * #5614 Enable run_job() for telemetry job * #5578 Fix on-insert decompression after schema changes * #5613 Quote username identifier appropriately * #5525 Fix tablespace for compressed hypertable and corresponding toast * #5642 Fix ALTER TABLE SET with normal tables * #5666 Reduce memory usage for distributed analyze * #5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

sb230132 self-assigned this Feb 16, 2023

sb230132 force-pushed the dml_support branch 4 times, most recently from d60d87c to 684f0e7 Compare February 17, 2023 17:27

sb230132 force-pushed the dml_support branch from 684f0e7 to f903c0d Compare February 18, 2023 10:20

sb230132 marked this pull request as ready for review February 20, 2023 04:21

github-actions bot requested review from fabriziomello and konskov February 20, 2023 04:22

svenklemm requested changes Feb 20, 2023

View reviewed changes

sb230132 force-pushed the dml_support branch from f903c0d to 4188039 Compare February 28, 2023 09:22

sb230132 changed the title ~~Support UPDATE/DELETE sql on compressed hypertables.~~ Support UPDATE/DELETE on compressed hypertables. Feb 28, 2023

sb230132 requested a review from svenklemm February 28, 2023 09:24

sb230132 force-pushed the dml_support branch 12 times, most recently from 7c198db to 0ac7790 Compare March 6, 2023 09:46

sb230132 mentioned this pull request Mar 6, 2023

Allow INSERTS/UPDATES/DELETES to compressed chunks #2857

Closed

sb230132 force-pushed the dml_support branch from 0ac7790 to 62d3ce5 Compare March 7, 2023 01:42

sb230132 force-pushed the dml_support branch from 6e12532 to b53d0b4 Compare April 4, 2023 08:19

sb230132 requested review from svenklemm and akuzm April 4, 2023 08:50

svenklemm reviewed Apr 4, 2023

View reviewed changes

src/nodes/hypertable_modify.c Outdated Show resolved Hide resolved

svenklemm reviewed Apr 4, 2023

View reviewed changes

src/nodes/hypertable_modify.h Outdated Show resolved Hide resolved

svenklemm reviewed Apr 4, 2023

View reviewed changes

tsl/src/compression/compression.c Outdated Show resolved Hide resolved

sb230132 force-pushed the dml_support branch 2 times, most recently from bf1d09c to 43b9b0c Compare April 4, 2023 11:57

sb230132 requested a review from svenklemm April 4, 2023 11:58

svenklemm reviewed Apr 5, 2023

View reviewed changes

tsl/src/compression/compression.c Outdated Show resolved Hide resolved

sb230132 closed this Apr 5, 2023

sb230132 force-pushed the dml_support branch from 43b9b0c to c2941a3 Compare April 5, 2023 10:42

svenklemm reopened this Apr 5, 2023

svenklemm reviewed Apr 5, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

svenklemm approved these changes Apr 5, 2023

View reviewed changes

sb230132 force-pushed the dml_support branch from 216b6b0 to 07d48d7 Compare April 5, 2023 11:18

sb230132 enabled auto-merge (rebase) April 5, 2023 11:23

sb230132 merged commit 1fb058b into timescale:main Apr 5, 2023

sb230132 deleted the dml_support branch April 19, 2023 04:57

kgyrtkirk mentioned this pull request May 17, 2023

Release 2.11.0 #5695

Merged

kgyrtkirk mentioned this pull request May 19, 2023

Release 2.11.0 - fix date #5702

Closed

mgagliardo91 mentioned this pull request Jun 30, 2023

[Feature]: Expose granular compression (row-level) #5844

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support UPDATE/DELETE on compressed hypertables. #5339

Support UPDATE/DELETE on compressed hypertables. #5339

sb230132 commented Feb 16, 2023 •

edited

Loading

codecov bot commented Feb 17, 2023 •

edited

Loading

github-actions bot commented Feb 20, 2023

svenklemm left a comment

sb230132 commented Feb 28, 2023

Micromegass commented May 16, 2023 •

edited

Loading

Support UPDATE/DELETE on compressed hypertables. #5339

Support UPDATE/DELETE on compressed hypertables. #5339

Conversation

sb230132 commented Feb 16, 2023 • edited Loading

codecov bot commented Feb 17, 2023 • edited Loading

Codecov Report

github-actions bot commented Feb 20, 2023

svenklemm left a comment

Choose a reason for hiding this comment

sb230132 commented Feb 28, 2023

Micromegass commented May 16, 2023 • edited Loading

sb230132 commented Feb 16, 2023 •

edited

Loading

codecov bot commented Feb 17, 2023 •

edited

Loading

Micromegass commented May 16, 2023 •

edited

Loading