Use a stateful detoaster for compressed data #6383
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #6383 +/- ##
==========================================
- Coverage 87.33% 87.27% -0.07%
==========================================
Files 184 185 +1
Lines 41591 41698 +107
Branches 9238 9263 +25
==========================================
+ Hits 36324 36392 +68
- Misses 3602 3626 +24
- Partials 1665 1680 +15 ☔ View full report in Codecov by Sentry. |
|
@antekresic, @mahipv: please review this pull request.
|
| @@ -1398,6 +1398,7 @@ tsl_recompress_chunk_segmentwise(PG_FUNCTION_ARGS) | |||
| index_endscan(index_scan); | |||
| UnregisterSnapshot(snapshot); | |||
| index_close(index_rel, AccessExclusiveLock); | |||
| row_decompressor_close(&decompressor); | |||
There was a problem hiding this comment.
Nit: we have row_compressor_finish and now row_decompressor_close which is kinda out of sync.
Also, maybe we could put more of the decompressor finish work in the function like freeing the estate, bistate, closing indexes and anything else necessary. If not in this PR, just add a TODO on the function.
There was a problem hiding this comment.
Nit: we have row_compressor_finish and now row_decompressor_close which is kinda out of sync.
I think "finish" might give a wrong impression that it finishes compression, so I renamed it to "close" as well.
Also, maybe we could put more of the decompressor finish work in the function like freeing the estate, bistate, closing indexes and anything else necessary. If not in this PR, just add a TODO on the function.
Makes sense, somehow I didn't notice them. I moved them into the "close" function as well.
| return query select sum(length(s1)) from longstr; | ||
| end; $$ language plpgsql volatile; | ||
|
|
||
| select sum(t) from generate_series(1, 30) x, lateral test(x * x * x) t; |
There was a problem hiding this comment.
For sake of coverage, lets decompress all the chunks using decompress_chunk as well. Maybe its enough to do it once after this select.
The normal Postgres detoasting code locks and opens toast tables and indexes for each toast value, which can take a large percentage CPU time on simple queries. Since in decompression we're working with one table at a time, the toast table and index are the same for every datum, so we don't have to redo this work.
The normal Postgres detoasting code locks and opens toast tables and indexes for each toast value, which can take a large percentage CPU time on simple queries. Since in decompression we're working with one table at a time, the toast table and index are the same for every datum, so we don't have to redo this work.
Gives up to 60% speedup on some queries to compressed hypertables: https://grafana.ops.savannah-dev.timescale.com/d/fasYic_4z/compare-akuzm?orgId=1&var-branch=All&var-run1=3012&var-run2=3018&var-threshold=0&var-use_historical_thresholds=true&var-threshold_expression=2.0%20*%20percentile_disc(0.90)&var-exact_suite_version=false
Disable-check: force-changelog-file