Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: _timescaledb_internal.create_compressed_chunk doesn't account for existing uncompressed rows #5946

Closed
JamesGuthrie opened this issue Aug 9, 2023 · 0 comments · Fixed by #5951
Closed

[Bug]: _timescaledb_internal.create_compressed_chunk doesn't account for existing uncompressed rows #5946

JamesGuthrie opened this issue Aug 9, 2023 · 0 comments · Fixed by #5951
Labels
bug

Comments

@JamesGuthrie
Copy link
Contributor

JamesGuthrie commented Aug 9, 2023 •
edited

What type of bug is this?

Incorrect result

What subsystems and features are affected?

Compression

What happened?

The _timescaledb_internal.create_compressed_chunk is an internal API which can be used to create a new compressed chunk using an existing table containing compressed data.

This function assumes that it will only be applied to a chunk which does not contain uncompressed rows. When the uncompressed chunk contains rows, they "disappear" after calling this function (i.e. the status in the timescaledb catalog is incorrectly set to 1 instead of 9 on the uncompressed chunk).

TimescaleDB version affected

2.11.1

PostgreSQL version used

15

What operating system did you use?

NixOS x64

What installation method did you use?

Docker

What platform did you run on?

Not applicable

Relevant log output and stack trace

No response

How can we reproduce the bug?

-- create compressed hypertable
CREATE TABLE "public"."metrics" (
    "time" timestamp with time zone NOT NULL,
    "device_id" "text",
    "val" double precision
);
SELECT create_hypertable('public.metrics', 'time');
ALTER TABLE metrics SET (timescaledb.compress, timescaledb.compress_orderby = 'time', timescaledb.compress_segmentby = 'device_id');

-- insert uncompressed rows into chunk (72 rows)
INSERT INTO metrics (time, device_id, val)
SELECT time, 1, random()
FROM generate_series('2023-05-01T00:00:00Z'::timestamptz, '2023-05-03T23:30:00Z'::timestamptz, '1 hour'::interval) time;

-- create table to contain compressed rows
CREATE TABLE "_timescaledb_internal"."bf_compress_hyper_2_6_chunk"() INHERITS ("_timescaledb_internal"."_compressed_hypertable_2");

-- insert compressed rows into compressed chunk (72 rows, compressed)
INSERT INTO "_timescaledb_internal"."bf_compress_hyper_2_6_chunk" (time, device_id, val, _ts_meta_count, _ts_meta_sequence_num, _ts_meta_min_1, _ts_meta_max_1)
  VALUES (
    'BAAAAp3QyLzcAAAAAADWk6QAAAAASAAAAAMAAAAAAAAP7gAFOyqLlMAAAAU7KN5td/8AAARgAAAAAA=='::_timescaledb_internal.compressed_data,
    1,
    'AwA/62ZV9sb6DgAAAEgAAAABAAAAAAAAAA8AAASAAAAAAQAAAEgAAAACAAAAAAAAAB8AAAQgAAAAAQAAAAAAAAAwAAAABxijCyyiSSjSgkwk0kkwooooKKKJJKOIIOKySSiijDCijEo0kkowo4ooMPJJLKKQKJIAAAAAACSSjAAAAEQAAAAHAAAAAAZmZmYMc89dsc8dfQ2yv2z01111DPHPTHffHSkNddM8ddc9NAz0z1v11x0yDHXW7XbXXLUAANstc87dswAAADsBv/Eu50tPulfM/5sfngH03RMKmj066bAxzKlCxP3+md6221EQnlRk2FC+4XEKqs4XH6QGMZFVhA3zXEe2x77lyuv6wv2sR7PoL3+j9nZ9DGNb4ffuo6LGw4qt7ES/tYgi88t4nBzLursm5A3eV5kGXx2o53V+kXAFW9E3F9/Ry+bn2p9hMI/iVCWafyuahm7g1A75tvLjPphErQDyPmQt2Ww+X1sseHIlY2bjczStN6K3aSD1f9I7+h+QPU8TZ2McO3bjhrQvA9/+H8GP2L/v49fvBvmf4zsHK+OhlLPgdaz+2NU0o/P47hlWktjxRvUHWuAk87ek3BSTNRq+/+aM/lTDxHTxg8F8Y/Px/BkI1+fZc2HJeP49qMFNpeL7Kv3LsC1lcsG7vcHkv/qUFTOjy6Qm5bNQB4Zqsf4QMfrhoIwbJIDaJrziLBh6GxSvSKtZnpsfRK8yy5TEbY33yeeHn5ZW9V7pa45IR6gher/TH0/cRfaL5P1nSy+hkK5Ooia4Rg8oYmXfLWH+luWPj78p1lhhr+Fo3sI7G6jl/7Id+dXMWGmvo9eFI/dAlPRROnk+meoSmMXehGEzxQgh7GLCy8qwJ+P3ETY/a7ApFnTd/woAAAAAAAAAAQ=='::_timescaledb_internal.compressed_data,
    72,
    10,
    '2023-05-01 00:00:00+00'::timestamptz,
    '2023-05-03 23:00:00+00'::timestamptz
  );

-- attach compressed chunk to parent chunk
SELECT _timescaledb_internal.create_compressed_chunk(
	    '"_timescaledb_internal"."_hyper_1_1_chunk"'::TEXT::REGCLASS,
	    '"_timescaledb_internal"."bf_compress_hyper_2_6_chunk"'::TEXT::REGCLASS,
	    8192, 8192,16384,8192,8192,16384,72,1);

-- select total rows in chunk, returns 72 (should be 144)
SELECT count(*) FROM "_timescaledb_internal"."_hyper_1_1_chunk";

-- rewrite catalog to include compressed chunk's rows
UPDATE _timescaledb_catalog.chunk SET status = 9 WHERE schema_name = '_timescaledb_internal' AND table_name = '_hyper_1_1_chunk';

-- select total rows in chunk, returns 144
SELECT count(*) FROM "_timescaledb_internal"."_hyper_1_1_chunk";
@JamesGuthrie JamesGuthrie changed the title [Bug]: _timescaledb_internal.create_compressed_chunk doesn't account for existing uncompressed rows [Bug]: _timescaledb_internal.create_compressed_chunk doesn't account for existing uncompressed rows Aug 9, 2023
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
386dcaa 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
@JamesGuthrie JamesGuthrie mentioned this issue Aug 10, 2023
Merged
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
461ff66 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
f9e15a0 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
3645b3e 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
db219cb 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
dbc2715 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
@JamesGuthrie JamesGuthrie mentioned this issue Aug 10, 2023
Closed
JamesGuthrie added a commit that referenced this issue Aug 10, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
b210b9f 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
04d432e 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
53dc0c8 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
c103228 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
0141b61 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
d45d402 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 23, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
1a4bc47 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 24, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
848749b 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 24, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
3fa53b7 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 24, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
a9672e2 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 24, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
3c70116 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
JamesGuthrie added a commit that referenced this issue Aug 28, 2023
@JamesGuthrie
Account for uncompressed rows in 'create_compressed_chunk'
01e480d 
`_timescaledb_internal.create_compressed_chunk` can be used to create a
compressed chunk with existing compressed data. It did not account for
the fact that the chunk can contain uncompressed data, in which case the
chunk status must be set to partial.

Fixes #5946
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Account for uncompressed rows in 'create_compressed_chunk' timescale/timescaledb
1 participant
@JamesGuthrie