New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor INSERT into compressed chunks #4926
Conversation
ea040be
to
9b97537
Compare
Depends on #4925 |
9b97537
to
16aa49c
Compare
16aa49c
to
b741330
Compare
5a0e71f
to
13f58ee
Compare
81145d1
to
bcb6a97
Compare
Codecov Report
@@ Coverage Diff @@
## main #4926 +/- ##
==========================================
- Coverage 89.61% 89.42% -0.19%
==========================================
Files 227 227
Lines 51744 51617 -127
==========================================
- Hits 46369 46160 -209
- Misses 5375 5457 +82
Continue to review full report at Codecov.
|
bcb6a97
to
0b1a900
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Please add compression-15.out
before merge it!
0b1a900
to
990a3b3
Compare
dd23a98
to
5e8b23f
Compare
sql/policy_internal.sql
Outdated
-- chunk status bits: | ||
-- 1: compressed | ||
-- 2: compressed unordered | ||
-- 4: frozen | ||
-- 8: compressed partial |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably need to update the high-level description of our compression logic. Do we have one somewhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm i dont think we have what you are asking here atm.
sql/policy_internal.sql
Outdated
@@ -65,6 +65,11 @@ BEGIN | |||
lag := _timescaledb_internal.subtract_integer_from_now(htoid, lag::BIGINT); | |||
END IF; | |||
|
|||
-- chunk status bits: | |||
-- 1: compressed | |||
-- 2: compressed unordered |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unordered chunks are not created by INSERTs anymore, but can still be created by UPDATEs, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We currently dont support UPDATE on compressed chunks. Unordered chunks won't be created by this INSERT change but can still be created by the merge chunk functionality of compression.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice simplification.
91bcb49
to
2ace43f
Compare
This patch changes INSERTs into compressed chunks to no longer be immediately compressed but stored in the uncompressed chunk instead and later merged with the compressed chunk by a separate job. This greatly simplifies the INSERT-codepath as we no longer have to rewrite the target of INSERTs and on-the-fly compress leading to a roughly 2x improvement on INSERT rate into compressed chunk. Additionally this improves TRIGGER-support for INSERTs into compressed chunks. This is a necessary refactoring to allow UPSERT/UPDATE/DELETE on compressed chunks in follow-patches.
2ace43f
to
b455357
Compare
Automated backport to 2.9.x not done: cherry-pick failed. |
This release contains new features and bug fixes since the 2.9.3 release. This release is high priority for upgrade. We strongly recommend that you upgrade as soon as possible. **Features** * #5241 Allow RETURNING clause when inserting into compressed chunks * #5245 Manage life-cycle of connections via memory contexts * #5246 Make connection establishment interruptible * #5253 Make data node command execution interruptible * #5243 Enable real-time aggregation for continuous aggregates with joins * #5262 Extend enabling compression on a continuous aggregrate with 'compress_segmentby' and 'compress_orderby' parameters **Bugfixes** * #4926 Fix corruption when inserting into compressed chunks * #5218 Add role-level security to job error log * #5214 Fix use of prepared statement in async module * #5290 Compression can't be enabled on continuous aggregates when segmentby/orderby columns need quotation * #5239 Fix next_start calculation for fixed schedules
This release contains new features and bug fixes since the 2.9.3 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Joins in continuous aggregates * Re-architecture of how compression works: ~2x improvement on INSERT rate into compressed chunks. * Full PostgreSQL 15 support for all existing features. Support for the newly introduced MERGE command on hypertables will be introduced on a follow-up release. **PostgreSQL 12 deprecation announcement** We will continue supporting PostgreSQL 12 until July 2023. Sooner to that time, we will announce the specific version of TimescaleDB in which PostgreSQL 12 support will not be included going forward. **Old format of Continuous Aggregates deprecation announcement** TimescaleDB 2.7 introduced a new format for continuous aggregates that improves performance. All instances with Continuous Aggregates using the old format should [migrate to the new format](https://docs.timescale.com/api/latest/continuous-aggregates/cagg_migrate/) by July 2023, when support for the old format will be removed. Sooner to that time, we will announce the specific version of TimescaleDB in which support for this feature will not be included going forward. **Features** * #4874 Allow joins in continuous aggregates * #4926 Refactor INSERT into compressed chunks * #5241 Allow RETURNING clause when inserting into compressed chunks * #5245 Manage life-cycle of connections via memory contexts * #5246 Make connection establishment interruptible * #5253 Make data node command execution interruptible * #5262 Extend enabling compression on a continuous aggregrate with 'compress_segmentby' and 'compress_orderby' parameters **Bugfixes** * #5214 Fix use of prepared statement in async module * #5218 Add role-level security to job error log * #5239 Fix next_start calculation for fixed schedules * #5290 Fix enabling compression on continuous aggregates with columns requiring quotation **Thanks** * @henriquegelio for reporting the issue on fixed schedules
This patch changes INSERTs into compressed chunks to no longer
be immediately compressed but stored in the uncompressed chunk
instead and later merged with the compressed chunk by a separate
job.
This greatly simplifies the INSERT-codepath as we no longer have
to rewrite the target of INSERTs and on-the-fly compress leading to a
roughly 2x improvement on INSERT rate into compressed chunk.
Additionally this improves TRIGGER-support for INSERTs into compressed chunks.
This is a necessary refactoring to allow UPSERT/UPDATE/DELETE on
compressed chunks in follow-patches.
Fixes #4655