Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reorder job fails because of already compressed chunks #1810

Closed
jocrau opened this issue Apr 10, 2020 · 1 comment · Fixed by #1854
Closed

Reorder job fails because of already compressed chunks #1810

jocrau opened this issue Apr 10, 2020 · 1 comment · Fixed by #1854
Labels
2.0-proposed Proposed for 2.0.0 release bug compression

Comments

@jocrau
Copy link

jocrau commented Apr 10, 2020

Relevant system information:

PostgreSQL 11.6 (Debian 11.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit

  • TimescaleDB version: 1.6.0
  • Deployed using TimescaleDB Helm charts

Describe the bug

We use the following compression policy:

ALTER TABLE timeseries.quantity SET (timescaledb.compress, timescaledb.compress_orderby = 'event_time, sequence', timescaledb.compress_segmentby = 'metric_ref');

for this table

create table quantity
(
	metric_ref integer not null,
	event_time timestamp not null,
	sequence integer not null,
	value double precision
);

create unique index quantity_pk
	on quantity (metric_ref asc, event_time desc, sequence desc);

Now we have added a

SELECT add_reorder_policy('timeseries.quantity', 'quantity_pk');

but the job fails with

2020-04-10 16:47:56 UTC | 29646 | 5e90a33c.73ce-2 | @ | 22023 | ERROR:  "_hyper_3_5237_chunk" is a compressed chunk
2020-04-10 16:47:56 UTC | 64 | 5e70f65c.40-103 | @ | 00000 | LOG:  background worker "Reorder Background Job" (PID 29646) exited with exit code 1

Expected behavior

We expected the job to finish by skipping already compressed chunks (as they cannot be changed and are already ordered).
@erimatnor erimatnor added compression 2.0-proposed Proposed for 2.0.0 release bug labels Apr 15, 2020
@jocrau
Copy link
Author

jocrau commented Apr 21, 2020

In the meantime, here is a workaround (specific to our domain model). The query marks all chunks as already reordered that are older than 28 days (our compression threshold). The reorder jobs have to be in place for this to work.

INSERT INTO _timescaledb_internal.bgw_policy_chunk_stats(job_id, chunk_id, num_times_job_run, last_time_job_run)
    SELECT job_id, chunk_id, 1 AS num_times_job_run, current_timestamp AS last_time_job_run
        FROM _timescaledb_catalog.chunk
        INNER JOIN (
                       SELECT *
                           FROM chunk_relation_size('quantity')
                       UNION
                       SELECT *
                           FROM chunk_relation_size('state')
                       UNION
                       SELECT *
                           FROM chunk_relation_size('info')
                       UNION
                       SELECT *
                           FROM chunk_relation_size('position')
                       UNION
                       SELECT *
                           FROM chunk_relation_size('counter')
                       UNION
                       SELECT *
                           FROM chunk_relation_size('observation')) crs ON chunk.id = crs.chunk_id
        INNER JOIN  _timescaledb_config.bgw_policy_reorder USING (hypertable_id)
        WHERE to_timestamp(upper(ranges[1]) / 1000000) < current_timestamp - INTERVAL '28 days';

Olernov added a commit to Olernov/timescaledb that referenced this issue Apr 26, 2020
@Olernov
Fix reorder policy job to skip compressed chunks
8b068ad 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes timescale#1810
@Olernov Olernov mentioned this issue Apr 26, 2020
Merged
Olernov added a commit to Olernov/timescaledb that referenced this issue Apr 26, 2020
@Olernov
Fix reorder policy job to skip compressed chunks
181e4bd 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes timescale#1810
Olernov added a commit to Olernov/timescaledb that referenced this issue May 6, 2020
@Olernov
Fix reorder policy job to skip compressed chunks
cac47ca 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes timescale#1810
Olernov added a commit to Olernov/timescaledb that referenced this issue May 6, 2020
@Olernov
Fix reorder policy job to skip compressed chunks
f0b7057 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes timescale#1810
Olernov added a commit to Olernov/timescaledb that referenced this issue May 7, 2020
@Olernov
Fix reorder policy job to skip compressed chunks
0fe95f2 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes timescale#1810
gayyappan pushed a commit that referenced this issue May 8, 2020
@Olernov @gayyappan
Fix reorder policy job to skip compressed chunks
b368563 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes #1810
svenklemm pushed a commit that referenced this issue May 13, 2020
@Olernov @svenklemm
Fix reorder policy job to skip compressed chunks
d9a4ded 
Reorder policy does not skip compressed chunks when selecting
next chunk to apply reordering. This causes an error in job
execution since it's not possible to reorder compressed chunks.
With this fix the job excludes compressed chunks from selection.

Fixes #1810
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
2.0-proposed Proposed for 2.0.0 release bug compression
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix reorder policy job to skip compressed chunks Olernov/timescaledb
2 participants
@jocrau @erimatnor