Enable indexscan on uncompressed part of partially compressed chunks #5542

konskov · 2023-04-11T10:04:07Z

This was previously disabled as no data resided on the
uncompressed chunk once it was compressed, but this is not
the case anymore with partially compressed chunks, so we
enable indexscan for the uncompressed chunk again.

Fixes #5432

Co-authored-by: Ante Kresic ante.kresic@gmail.com

codecov · 2023-04-11T10:18:34Z

Codecov Report

Merging #5542 (ca8ae06) into main (ca472eb) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #5542      +/-   ##
==========================================
- Coverage   90.56%   90.41%   -0.15%     
==========================================
  Files         229      229              
  Lines       47497    53994    +6497     
==========================================
+ Hits        43015    48821    +5806     
- Misses       4482     5173     +691

Impacted Files	Coverage Δ
src/planner/planner.c	`90.60% <100.00%> (-0.54%)`	⬇️

... and 203 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-04-13T06:44:58Z

@gayyappan, @fabriziomello: please review this pull request.

Powered by pull-review

src/planner/planner.c

svenklemm · 2023-04-14T12:42:06Z

tsl/test/expected/compression_ddl.out

 SELECT device_id, count(*)
 FROM compression_insert
 GROUP BY device_id;


Those queries are missing ORDER BY clause making them indeterministic and potentially leading to flaky tests.

Oh sorry I just realized that I did not address this comment... :( I will create a follow-up PR for this

This was previously disabled as no data resided on the uncompressed chunk once it was compressed, but this is not the case anymore with partially compressed chunks, so we enable indexscan for the uncompressed chunk again. Fixes timescale#5432 Co-authored-by: Ante Kresic <ante.kresic@gmail.com>

timescale-automation · 2023-04-19T08:20:40Z

Automated backport to 2.10.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.10.x
You are currently cherry-picking commit 5633960f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.md

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   src/planner/planner.c
	both modified:   tsl/test/expected/compression_ddl.out
	both modified:   tsl/test/sql/compression_ddl.sql

Job log

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * timescale#5410 Fix file trailer handling in the COPY fetcher * timescale#5446 Add checks for malloc failure in libpq calls * timescale#5233 Out of on_proc_exit slots on guc license change * timescale#5428 Use consistent snapshots when scanning metadata * timescale#5499 Do not segfault on large histogram() parameters * timescale#5470 Ensure superuser perms during copy/move chunk * timescale#5500 Fix when no FROM clause in continuous aggregate definition * timescale#5433 Fix join rte in CAggs with joins * timescale#5556 Fix duplicated entries on timescaledb_experimental.policies view * timescale#5462 Fix segfault after column drop on compressed table * timescale#5543 Copy scheduled_jobs list before sorting it * timescale#5497 Allow named time_bucket arguments in Cagg definition * timescale#5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * timescale#5558 Use regrole for job owner * timescale#5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * #5410 Fix file trailer handling in the COPY fetcher * #5446 Add checks for malloc failure in libpq calls * #5233 Out of on_proc_exit slots on guc license change * #5428 Use consistent snapshots when scanning metadata * #5499 Do not segfault on large histogram() parameters * #5470 Ensure superuser perms during copy/move chunk * #5500 Fix when no FROM clause in continuous aggregate definition * #5433 Fix join rte in CAggs with joins * #5556 Fix duplicated entries on timescaledb_experimental.policies view * #5462 Fix segfault after column drop on compressed table * #5543 Copy scheduled_jobs list before sorting it * #5497 Allow named time_bucket arguments in Cagg definition * #5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * #5558 Use regrole for job owner * #5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * #5410 Fix file trailer handling in the COPY fetcher * #5446 Add checks for malloc failure in libpq calls * #5233 Out of on_proc_exit slots on guc license change * #5428 Use consistent snapshots when scanning metadata * #5499 Do not segfault on large histogram() parameters * #5470 Ensure superuser perms during copy/move chunk * #5500 Fix when no FROM clause in continuous aggregate definition * #5433 Fix join rte in CAggs with joins * #5556 Fix duplicated entries on timescaledb_experimental.policies view * #5462 Fix segfault after column drop on compressed table * #5543 Copy scheduled_jobs list before sorting it * #5497 Allow named time_bucket arguments in Cagg definition * #5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * #5558 Use regrole for job owner * #5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * #5410 Fix file trailer handling in the COPY fetcher * #5446 Add checks for malloc failure in libpq calls * #5233 Out of on_proc_exit slots on guc license change * #5428 Use consistent snapshots when scanning metadata * #5499 Do not segfault on large histogram() parameters * #5470 Ensure superuser perms during copy/move chunk * #5500 Fix when no FROM clause in continuous aggregate definition * #5433 Fix join rte in CAggs with joins * #5556 Fix duplicated entries on timescaledb_experimental.policies view * #5462 Fix segfault after column drop on compressed table * #5543 Copy scheduled_jobs list before sorting it * #5497 Allow named time_bucket arguments in Cagg definition * #5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * #5558 Use regrole for job owner * #5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * timescale#5410 Fix file trailer handling in the COPY fetcher * timescale#5446 Add checks for malloc failure in libpq calls * timescale#5233 Out of on_proc_exit slots on guc license change * timescale#5428 Use consistent snapshots when scanning metadata * timescale#5499 Do not segfault on large histogram() parameters * timescale#5470 Ensure superuser perms during copy/move chunk * timescale#5500 Fix when no FROM clause in continuous aggregate definition * timescale#5433 Fix join rte in CAggs with joins * timescale#5556 Fix duplicated entries on timescaledb_experimental.policies view * timescale#5462 Fix segfault after column drop on compressed table * timescale#5543 Copy scheduled_jobs list before sorting it * timescale#5497 Allow named time_bucket arguments in Cagg definition * timescale#5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * timescale#5558 Use regrole for job owner * timescale#5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * #5410 Fix file trailer handling in the COPY fetcher * #5446 Add checks for malloc failure in libpq calls * #5233 Out of on_proc_exit slots on guc license change * #5428 Use consistent snapshots when scanning metadata * #5499 Do not segfault on large histogram() parameters * #5470 Ensure superuser perms during copy/move chunk * #5500 Fix when no FROM clause in continuous aggregate definition * #5433 Fix join rte in CAggs with joins * #5556 Fix duplicated entries on timescaledb_experimental.policies view * #5462 Fix segfault after column drop on compressed table * #5543 Copy scheduled_jobs list before sorting it * #5497 Allow named time_bucket arguments in Cagg definition * #5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * #5558 Use regrole for job owner * #5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

@nikolaps

## 2.10.2 (2023-04-20) **Bugfixes** * #5410 Fix file trailer handling in the COPY fetcher * #5446 Add checks for malloc failure in libpq calls * #5233 Out of on_proc_exit slots on guc license change * #5428 Use consistent snapshots when scanning metadata * #5499 Do not segfault on large histogram() parameters * #5470 Ensure superuser perms during copy/move chunk * #5500 Fix when no FROM clause in continuous aggregate definition * #5433 Fix join rte in CAggs with joins * #5556 Fix duplicated entries on timescaledb_experimental.policies view * #5462 Fix segfault after column drop on compressed table * #5543 Copy scheduled_jobs list before sorting it * #5497 Allow named time_bucket arguments in Cagg definition * #5544 Fix refresh from beginning of Continuous Aggregate with variable time bucket * #5558 Use regrole for job owner * #5542 Enable indexscan on uncompressed part of partially compressed chunks **Thanks** * @nikolaps for reporting an issue with the COPY fetcher * @S-imo-n for reporting the issue on Background Worker Scheduler crash * @geezhu for reporting issue on segfault in historgram() * @mwahlhuetter for reporting the issue with joins in CAggs * @mwahlhuetter for reporting issue with duplicated entries on timescaledb_experimental.policies view * @H25E for reporting error refreshing from beginning of a Continuous Aggregate with variable time bucket

github-actions bot assigned konskov Apr 11, 2023

konskov changed the title ~~Ante fix indexscan partial 5432~~ Enable index scan on uncompressed part of partially compressed chunks Apr 11, 2023

konskov force-pushed the ante-fix-indexscan-partial-5432 branch 3 times, most recently from d9b8681 to d724a35 Compare April 13, 2023 06:42

konskov requested a review from antekresic April 13, 2023 06:44

konskov marked this pull request as ready for review April 13, 2023 06:44

github-actions bot requested review from fabriziomello and gayyappan April 13, 2023 06:44

svenklemm reviewed Apr 13, 2023

View reviewed changes

src/planner/planner.c Outdated Show resolved Hide resolved

konskov force-pushed the ante-fix-indexscan-partial-5432 branch from d724a35 to 5ff118e Compare April 13, 2023 12:50

antekresic approved these changes Apr 13, 2023

View reviewed changes

konskov requested a review from svenklemm April 13, 2023 13:10

konskov changed the title ~~Enable index scan on uncompressed part of partially compressed chunks~~ Enable indexscan on uncompressed part of partially compressed chunks Apr 13, 2023

svenklemm reviewed Apr 14, 2023

View reviewed changes

svenklemm approved these changes Apr 14, 2023

View reviewed changes

konskov force-pushed the ante-fix-indexscan-partial-5432 branch from 5ff118e to 1dc65df Compare April 18, 2023 13:31

konskov force-pushed the ante-fix-indexscan-partial-5432 branch from 1dc65df to ca8ae06 Compare April 18, 2023 14:00

konskov merged commit 5633960 into timescale:main Apr 18, 2023
47 of 48 checks passed

timescale-automation added the auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) label Apr 19, 2023

timescale-automation added the backported-2.10.x label Apr 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable indexscan on uncompressed part of partially compressed chunks #5542

Enable indexscan on uncompressed part of partially compressed chunks #5542

konskov commented Apr 11, 2023 •

edited

codecov bot commented Apr 11, 2023 •

edited

github-actions bot commented Apr 13, 2023

svenklemm Apr 14, 2023

konskov Apr 18, 2023

timescale-automation commented Apr 19, 2023

Enable indexscan on uncompressed part of partially compressed chunks #5542

Enable indexscan on uncompressed part of partially compressed chunks #5542

Conversation

konskov commented Apr 11, 2023 • edited

codecov bot commented Apr 11, 2023 • edited

Codecov Report

github-actions bot commented Apr 13, 2023

svenklemm Apr 14, 2023

Choose a reason for hiding this comment

konskov Apr 18, 2023

Choose a reason for hiding this comment

timescale-automation commented Apr 19, 2023

Git status

konskov commented Apr 11, 2023 •

edited

codecov bot commented Apr 11, 2023 •

edited