Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stats improvement for Uncompressed Chunks #5118

Merged
merged 1 commit into from Mar 22, 2023

Conversation

shhnwz
Copy link
Contributor

@shhnwz shhnwz commented Dec 23, 2022

During the compression autovacuum use to be disabled for uncompressed chunk and enable after decompression. This leads to postgres maintainence issue. Let's not disable autovacuum for uncompressed chunk anymore. Let postgres take care of the stats in its natural way.

Fixes #3376 #4578 #4156

@codecov
Copy link

codecov bot commented Jan 2, 2023

Codecov Report

Merging #5118 (66e4bdf) into main (5c07a57) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

❗ Current head 66e4bdf differs from pull request most recent head 6da5da4. Consider uploading reports for the commit 6da5da4 to get more accurate results

@@            Coverage Diff             @@
##             main    #5118      +/-   ##
==========================================
- Coverage   90.70%   90.67%   -0.03%     
==========================================
  Files         228      228              
  Lines       53106    52981     -125     
==========================================
- Hits        48168    48040     -128     
- Misses       4938     4941       +3     
Impacted Files Coverage Δ
src/cross_module_fn.c 67.91% <ø> (+1.07%) ⬆️
src/planner/planner.c 95.98% <ø> (+0.09%) ⬆️
tsl/src/compression/compression.c 96.11% <ø> (+0.28%) ⬆️
tsl/src/compression/compression.h 0.00% <ø> (ø)
tsl/src/init.c 96.29% <ø> (ø)
src/process_utility.c 94.43% <100.00%> (-0.10%) ⬇️
tsl/src/compression/api.c 95.60% <100.00%> (-0.22%) ⬇️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@shhnwz shhnwz marked this pull request as ready for review January 2, 2023 10:02
@shhnwz shhnwz self-assigned this Jan 2, 2023
@shhnwz shhnwz requested review from svenklemm and akuzm January 2, 2023 10:06
@shhnwz shhnwz enabled auto-merge (rebase) January 3, 2023 08:59
@akuzm
Copy link
Member

akuzm commented Jan 4, 2023

The issue also says:

we should instead use the get_relation_stats_hook to handle compressed chunk stats.

Do we have to change it in this PR as well?

@shhnwz shhnwz requested a review from akuzm January 6, 2023 05:19
@shhnwz shhnwz marked this pull request as draft January 25, 2023 13:51
auto-merge was automatically disabled January 25, 2023 13:51

Pull request was converted to draft

@shhnwz shhnwz force-pushed the 309-improve-stats-handling branch 12 times, most recently from 8445d94 to ac6b54b Compare February 1, 2023 13:52
@shhnwz shhnwz force-pushed the 309-improve-stats-handling branch 4 times, most recently from db715f9 to b94e7ef Compare February 2, 2023 09:09
@shhnwz shhnwz force-pushed the 309-improve-stats-handling branch from 66e4bdf to 6da5da4 Compare March 22, 2023 14:27
@shhnwz shhnwz merged commit 699fcf4 into timescale:main Mar 22, 2023
46 of 47 checks passed
@shhnwz shhnwz deleted the 309-improve-stats-handling branch March 22, 2023 18:21
@timescale-automation
Copy link

Automated backport to 2.10.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.10.x
You are currently cherry-picking commit 699fcf48.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .github/gh_matrix_builder.py
	modified:   .github/workflows/windows-build-and-test.yaml
	modified:   CHANGELOG.md
	modified:   sql/size_utils.sql
	modified:   src/cross_module_fn.c
	modified:   src/cross_module_fn.h
	modified:   src/planner/planner.c
	modified:   src/process_utility.c
	modified:   tsl/src/compression/api.c
	modified:   tsl/src/compression/compression.h
	modified:   tsl/src/init.c
	modified:   tsl/test/expected/chunk_merge.out
	modified:   tsl/test/expected/telemetry_stats-12.out
	modified:   tsl/test/expected/telemetry_stats-13.out
	modified:   tsl/test/expected/telemetry_stats-14.out
	modified:   tsl/test/expected/telemetry_stats-15.out
	modified:   tsl/test/expected/transparent_decompression-12.out
	modified:   tsl/test/expected/transparent_decompression-14.out
	modified:   tsl/test/expected/transparent_decompression-15.out
	modified:   tsl/test/isolation/expected/compression_chunk_race.out
	modified:   tsl/test/shared/expected/extension.out
	modified:   tsl/test/shared/expected/ordered_append_join-12.out
	modified:   tsl/test/shared/expected/ordered_append_join-13.out
	modified:   tsl/test/shared/expected/ordered_append_join-14.out
	modified:   tsl/test/shared/expected/ordered_append_join-15.out
	modified:   tsl/test/shared/expected/transparent_decompress_chunk-12.out
	modified:   tsl/test/shared/expected/transparent_decompress_chunk-13.out
	modified:   tsl/test/shared/expected/transparent_decompress_chunk-14.out
	modified:   tsl/test/shared/expected/transparent_decompress_chunk-15.out
	modified:   tsl/test/sql/chunk_merge.sql
	modified:   tsl/test/sql/compression.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   sql/updates/reverse-dev.sql
	both modified:   tsl/src/compression/compression.c
	both modified:   tsl/test/expected/compression.out
	both modified:   tsl/test/expected/transparent_decompression-13.out


Job log

@timescale-automation timescale-automation added the auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) label Mar 22, 2023
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 12, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 13, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 14, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 17, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 17, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 17, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 17, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
shhnwz pushed a commit to shhnwz/timescaledb that referenced this pull request Apr 18, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of timescale#5118. This issue is to fix the flaky nature of
the test.
@svenklemm svenklemm mentioned this pull request Apr 18, 2023
shhnwz pushed a commit that referenced this pull request Apr 18, 2023
Transparent decompress chunk was added into to ignore list due to
the side effect of #5118. This issue is to fix the flaky nature of
the test.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to enable autovacuum for existing chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to enable autovacuum for existing chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum for existing
chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum for existing
chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum for existing
chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum for existing
chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum setting for
existing chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum setting for
existing chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum setting for
existing chunks.
jnidzwetzki added a commit to jnidzwetzki/timescaledb that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in timescale#5118, but
no migration was performed to reset the custom autovacuum setting for
existing chunks.
jnidzwetzki added a commit that referenced this pull request Mar 8, 2024
In old TSDB versions, we disabled autovacuum for compressed chunks to
keep the statistics. However, this restriction was removed in #5118, but
no migration was performed to reset the custom autovacuum setting for
existing chunks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) Query Experience Team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Size of the chunks increase without being decompressed if decompress_chunk(i) fails in middle
5 participants