Support ON CONFLICT DO NOTHING on compressed hypertables #5252

svenklemm · 2023-01-30T18:17:25Z

This patch allows unique constraints on compressed chunks. When trying to INSERT into compressed chunks with unique constraints any potentially conflicting compressed batches will be decompressed to let postgres do constraint checking on the INSERT. With this patch only INSERT ON CONFLICT DO NOTHING will be supported. For decompression only segment by information is considered to determine conflicting batches. This will be enhanced in a follow-up patch to also include orderby metadata to require decompressing less batches.

codecov · 2023-02-09T07:55:49Z

Codecov Report

Merging #5252 (db626a9) into main (c02cb76) will increase coverage by 0.00%.
The diff coverage is 96.07%.

@@           Coverage Diff           @@
##             main    #5252   +/-   ##
=======================================
  Coverage   90.71%   90.71%           
=======================================
  Files         227      227           
  Lines       52756    52831   +75     
=======================================
+ Hits        47857    47927   +70     
- Misses       4899     4904    +5

Impacted Files	Coverage Δ
src/chunk.h	`100.00% <ø> (ø)`
src/indexing.c	`95.21% <ø> (ø)`
tsl/src/compression/compression.h	`0.00% <ø> (ø)`
tsl/src/init.c	`96.29% <ø> (ø)`
src/nodes/chunk_dispatch/chunk_dispatch.c	`96.34% <77.77%> (-0.48%)`	⬇️
tsl/src/compression/compression.c	`95.81% <97.64%> (+0.24%)`	⬆️
src/chunk.c	`93.74% <100.00%> (+<0.01%)`	⬆️
src/copy.c	`94.70% <100.00%> (+0.01%)`	⬆️
src/nodes/chunk_dispatch/chunk_insert_state.c	`97.89% <100.00%> (+0.31%)`	⬆️
src/process_utility.c	`94.52% <100.00%> (+0.03%)`	⬆️
... and 1 more

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-02-09T09:23:15Z

@erimatnor, @jnidzwetzki: please review this pull request.

Powered by pull-review

src/nodes/chunk_dispatch/chunk_dispatch.c

tsl/src/compression/compression.c

akuzm · 2023-02-09T10:01:49Z

tsl/src/compression/compression.c

+		build_scankeys(chunk->fd.hypertable_id, decompressor, unique_columns, slot, &num_scankeys);
+
+	TableScanDesc heapScan =
+		table_beginscan(in_rel, GetTransactionSnapshot(), num_scankeys, scankeys);


This is a seq scan, probably we want to have an index scan?

We need to support both, you cant have scankeys on non-index column in indexscan, right?
But indexscan support will not be part of initial PR

Right. But Postgres requires there to be an index for the ON CONFLICT columns, so we should have one. I think it's OK with seq scan for starters, but performance won't be good and we probably need to add index scan soon.

Oh, it's more difficult with orderby columns, we don't have min/max in the default index for them.

the index will be on the uncompressed part though, not the compressed part

Let's discuss the indexes a little more. I think the implementation with seq scan is OK to merge for starters, but it's not something we should publish. Doing a seq scan for each inserted row is going to be prohibitively slow.

I think it is reasonable for us to require an index on the decompressed chunk that matches the ON CONFLICT clause. It's the same thing that Postgres requires.

We already create the index on segmentby columns, so it always can be used.

For min/max, can probably use a seq scan?

Ideally, if Postgres managed to find an arbiter index on the uncompressed chunk, we should have a matching index on the decompressed chunk. Not sure if we do anything to guarantee that currently.

Do we have a plan for this somewhere?

There will never be a matching index if you have unique constraints. Since unique constraints need to include the partitioning columns that means time will be part of the unique constraint but you will never want time as segmentby .

tsl/src/compression/compression.c

konskov · 2023-03-13T10:10:07Z

approving, I think this should be 1 commit though, right?

This patch allows unique constraints on compressed chunks. When trying to INSERT into compressed chunks with unique constraints any potentially conflicting compressed batches will be decompressed to let postgres do constraint checking on the INSERT. With this patch only INSERT ON CONFLICT DO NOTHING will be supported. For decompression only segment by information is considered to determine conflicting batches. This will be enhanced in a follow-up patch to also include orderby metadata to require decompressing less batches.

@kovetskiy

This release includes these noteworthy features: * compressed hypertable enhancements: * UPDATE/DELETE support * ON CONFLICT DO UPDATE * Join support for hierarchical Continougs Aggregates * performance improvements **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * #5212 Allow pushdown of reference table joins * #5221 Improve Realtime Continuous Aggregate performance * #5252 Improve unique constraint support on compressed hypertables * #5339 Support UPDATE/DELETE on compressed hypertables * #5344 Enable JOINS for Hierarchical Continuous Aggregates * #5361 Add parallel support for partialize_agg() * #5417 Refactor and optimize distributed COPY * #5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * #5547 Skip Ordered Append when only 1 child node is present * #5510 Propagate vacuum/analyze to compressed chunks * #5584 Reduce decompression during constraint checking * #5530 Optimize compressed chunk resorting * #5639 Support sending telemetry event reports **Bugfixes** * #5396 Fix SEGMENTBY columns predicates to be pushed down * #5427 Handle user-defined FDW options properly * #5442 Decompression may have lost DEFAULT values * #5459 Fix issue creating dimensional constraints * #5570 Improve interpolate error message on datatype mismatch * #5573 Fix unique constraint on compressed tables * #5615 Add permission checks to run_job() * #5614 Enable run_job() for telemetry job * #5578 Fix on-insert decompression after schema changes * #5613 Quote username identifier appropriately * #5525 Fix tablespace for compressed hypertable and corresponding toast * #5642 Fix ALTER TABLE SET with normal tables * #5666 Reduce memory usage for distributed analyze * #5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

@kovetskiy

This release contains new features and bug fixes since the 2.10.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Support for DML operations on compressed chunks: * UPDATE/DELETE support * Support for unique constraints on compressed chunks * Support for `ON CONFLICT DO UPDATE` * Support for `ON CONFLICT DO NOTHING` * Join support for hierarchical Continuous Aggregates **Features** * timescale#5212 Allow pushdown of reference table joins * timescale#5221 Improve Realtime Continuous Aggregate performance * timescale#5252 Improve unique constraint support on compressed hypertables * timescale#5339 Support UPDATE/DELETE on compressed hypertables * timescale#5344 Enable JOINS for Hierarchical Continuous Aggregates * timescale#5361 Add parallel support for partialize_agg() * timescale#5417 Refactor and optimize distributed COPY * timescale#5454 Add support for ON CONFLICT DO UPDATE for compressed hypertables * timescale#5547 Skip Ordered Append when only 1 child node is present * timescale#5510 Propagate vacuum/analyze to compressed chunks * timescale#5584 Reduce decompression during constraint checking * timescale#5530 Optimize compressed chunk resorting * timescale#5639 Support sending telemetry event reports **Bugfixes** * timescale#5396 Fix SEGMENTBY columns predicates to be pushed down * timescale#5427 Handle user-defined FDW options properly * timescale#5442 Decompression may have lost DEFAULT values * timescale#5459 Fix issue creating dimensional constraints * timescale#5570 Improve interpolate error message on datatype mismatch * timescale#5573 Fix unique constraint on compressed tables * timescale#5615 Add permission checks to run_job() * timescale#5614 Enable run_job() for telemetry job * timescale#5578 Fix on-insert decompression after schema changes * timescale#5613 Quote username identifier appropriately * timescale#5525 Fix tablespace for compressed hypertable and corresponding toast * timescale#5642 Fix ALTER TABLE SET with normal tables * timescale#5666 Reduce memory usage for distributed analyze * timescale#5668 Fix subtransaction resource owner **Thanks** * @kovetskiy and @DZDomi for reporting peformance regression in Realtime Continuous Aggregates * @ollz272 for reporting an issue with interpolate error messages

github-actions bot assigned svenklemm Jan 30, 2023

svenklemm changed the title ~~Support unique constraints on compressed chunks~~ WIP Support unique constraints on compressed chunks Jan 30, 2023

svenklemm force-pushed the compression_conflict_nothing branch 6 times, most recently from c337542 to 75d0b23 Compare January 31, 2023 06:21

svenklemm added this to the TimescaleDB 2.10 milestone Jan 31, 2023

svenklemm force-pushed the compression_conflict_nothing branch from 75d0b23 to c695f44 Compare February 1, 2023 16:34

svenklemm removed this from the TimescaleDB 2.10 milestone Feb 5, 2023

svenklemm force-pushed the compression_conflict_nothing branch 5 times, most recently from 85316dc to dd49732 Compare February 9, 2023 07:40

svenklemm changed the title ~~WIP Support unique constraints on compressed chunks~~ Support unique constraints on compressed chunks Feb 9, 2023

svenklemm force-pushed the compression_conflict_nothing branch from dd49732 to 07b601f Compare February 9, 2023 09:10

svenklemm marked this pull request as ready for review February 9, 2023 09:22

github-actions bot requested review from erimatnor and jnidzwetzki February 9, 2023 09:23

akuzm reviewed Feb 9, 2023

View reviewed changes

src/nodes/chunk_dispatch/chunk_dispatch.c Outdated Show resolved Hide resolved

akuzm reviewed Feb 9, 2023

View reviewed changes

tsl/src/compression/compression.c Outdated Show resolved Hide resolved

akuzm reviewed Feb 9, 2023

View reviewed changes

tsl/src/compression/compression.c Show resolved Hide resolved

akuzm reviewed Feb 9, 2023

View reviewed changes

tsl/src/compression/compression.c Show resolved Hide resolved

svenklemm force-pushed the compression_conflict_nothing branch 2 times, most recently from 9c3eb30 to 960e330 Compare February 9, 2023 16:59

sb230132 mentioned this pull request Mar 6, 2023

Support 'ON CONFLICT DO NOTHING' for compressed chunks #4113

Closed

svenklemm force-pushed the compression_conflict_nothing branch 10 times, most recently from dcfa92d to bf50033 Compare March 12, 2023 19:25

svenklemm requested review from konskov and antekresic March 13, 2023 09:59

konskov approved these changes Mar 13, 2023

View reviewed changes

svenklemm force-pushed the compression_conflict_nothing branch 2 times, most recently from 286086c to 87d73e9 Compare March 13, 2023 10:28

svenklemm force-pushed the compression_conflict_nothing branch from 87d73e9 to db626a9 Compare March 13, 2023 10:36

svenklemm merged commit 65562f0 into timescale:main Mar 13, 2023

jfjoly added this to the TimescaleDB 2.11 milestone Mar 14, 2023

riyazkittur mentioned this pull request Mar 16, 2023

[Bug]: Continuous aggregate is not refreshed automatically after insert into compressed chunks #5455

Closed

kgyrtkirk mentioned this pull request May 17, 2023

Release 2.11.0 #5695

Merged

kgyrtkirk mentioned this pull request May 19, 2023

Release 2.11.0 - fix date #5702

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support ON CONFLICT DO NOTHING on compressed hypertables #5252

Support ON CONFLICT DO NOTHING on compressed hypertables #5252

svenklemm commented Jan 30, 2023

codecov bot commented Feb 9, 2023 •

edited

Loading

github-actions bot commented Feb 9, 2023

akuzm Feb 9, 2023

svenklemm Feb 9, 2023

akuzm Feb 9, 2023

akuzm Feb 9, 2023

svenklemm Feb 9, 2023

akuzm Feb 14, 2023

svenklemm Feb 14, 2023 •

edited

Loading

konskov commented Mar 13, 2023

Support ON CONFLICT DO NOTHING on compressed hypertables #5252

Support ON CONFLICT DO NOTHING on compressed hypertables #5252

Conversation

svenklemm commented Jan 30, 2023

codecov bot commented Feb 9, 2023 • edited Loading

Codecov Report

github-actions bot commented Feb 9, 2023

akuzm Feb 9, 2023

Choose a reason for hiding this comment

svenklemm Feb 9, 2023

Choose a reason for hiding this comment

akuzm Feb 9, 2023

Choose a reason for hiding this comment

akuzm Feb 9, 2023

Choose a reason for hiding this comment

svenklemm Feb 9, 2023

Choose a reason for hiding this comment

akuzm Feb 14, 2023

Choose a reason for hiding this comment

svenklemm Feb 14, 2023 • edited Loading

Choose a reason for hiding this comment

konskov commented Mar 13, 2023

codecov bot commented Feb 9, 2023 •

edited

Loading

svenklemm Feb 14, 2023 •

edited

Loading