Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rename add_drop_chunks_policy #2148

Merged
merged 1 commit into from Jul 30, 2020
Merged

Rename add_drop_chunks_policy #2148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 14 additions & 5 deletions sql/bgw_scheduler.sql
View file
Open in desktop
Expand Up @@ -21,13 +21,21 @@ INSERT INTO _timescaledb_config.bgw_job (id, application_name, job_type, schedul
(1, 'Telemetry Reporter', 'telemetry_and_version_check_if_enabled', INTERVAL '24h', INTERVAL '100s', -1, INTERVAL '1h')
ON CONFLICT (id) DO NOTHING;

CREATE OR REPLACE FUNCTION add_drop_chunks_policy(
-- Add a retention policy to a hypertable or continuous aggregate.
-- The retention_window (typically an INTERVAL) determines the
-- window beyond which data is dropped at the time
-- of execution of the policy (e.g., '1 week'). Note that the retention
-- window will always align with chunk boundaries, thus the window
-- might be larger than the given one, but never smaller. In other
-- words, some data beyond the retention window
-- might be kept, but data within the window will never be deleted.
Comment on lines +24 to +31
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (on my own suggestion):

Suggested change
-- Add a retention policy to a hypertable or continuous aggregate.
-- The retention_window (typically an INTERVAL) determines the
-- window beyond which data is dropped at the time
-- of execution of the policy (e.g., '1 week'). Note that the retention
-- window will always align with chunk boundaries, thus the window
-- might be larger than the given one, but never smaller. In other
-- words, some data beyond the retention window
-- might be kept, but data within the window will never be deleted.
-- Add a retention policy to a hypertable or continuous aggregate.
-- The retention_window (typically an INTERVAL) determines the
-- window (e.g., '1 week') beyond which data is dropped at the time
-- of execution of the policy . Note that the retention
-- window will always align with chunk boundaries, thus the window
-- might be larger than the given one, but never smaller. In other
-- words, some data beyond the retention window
-- might be kept, but data within the window will never be deleted.

CREATE OR REPLACE FUNCTION add_retention_policy(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should take the opportunity to also add a description of this function:

Suggested change
CREATE OR REPLACE FUNCTION add_retention_policy(
-- Add a retention policy to a hypertable or continuous aggregate.
-- The retention_window (typically an INTERVAL) determines the
-- window beyond which data is dropped at the time
-- of execution of the policy (e.g., '1 week'). Note that the retention
-- window will always align with chunk boundaries, thus the window
-- might be larger than the given one, but never smaller. In other
-- words, some data beyond the retention window
-- might be kept, but data within the window will never be deleted.
CREATE OR REPLACE FUNCTION add_retention_policy(

Looking at this signature, it is unclear what the time epoch is from here the retention window starts. I assume it is wall clock (system clock), but I am not sure that always fits the data. We might need to add a separate parameter that allows the user to specify a function that gives the epoch, typically defaulting to now(), similar to what I proposed for the add_refresh_policy function. This is obviously for another PR, however.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I see the case where one wouldn't assume this was wall clock?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably not the common case, but if the data ingested is not aligned with the wall clock, then the retention policy won't work. Let's say, for instance, you are importing data in TIMESTAMP WITHOUT TIMEZONE and the database runs in another timezone than where the data originated.

Or, let's say your data uses some completely different notion of time, e.g., an integer that is an increasing counter, e.g., event number. The question is if the retention internally uses the integer_now_function in that case to get the current notion of time when the policy runs?

hypertable REGCLASS,
older_than "any",
retention_window "any",
if_not_exists BOOL = false,
cascade_to_materializations BOOL = false
)
RETURNS INTEGER AS '@MODULE_PATHNAME@', 'ts_add_drop_chunks_policy'
RETURNS INTEGER AS '@MODULE_PATHNAME@', 'ts_add_retention_policy'
LANGUAGE C VOLATILE STRICT;

CREATE OR REPLACE FUNCTION add_reorder_policy(hypertable REGCLASS, index_name NAME, if_not_exists BOOL = false) RETURNS INTEGER
Expand All @@ -47,8 +55,9 @@ RETURNS INTEGER
AS '@MODULE_PATHNAME@', 'ts_add_compress_chunks_policy'
LANGUAGE C VOLATILE STRICT;

CREATE OR REPLACE FUNCTION remove_drop_chunks_policy(hypertable REGCLASS, if_exists BOOL = false) RETURNS VOID
AS '@MODULE_PATHNAME@', 'ts_remove_drop_chunks_policy'
-- Remove the retention policy from a hypertable
CREATE OR REPLACE FUNCTION remove_retention_policy(hypertable REGCLASS, if_exists BOOL = false) RETURNS VOID
AS '@MODULE_PATHNAME@', 'ts_remove_retention_policy'
LANGUAGE C VOLATILE STRICT;

CREATE OR REPLACE FUNCTION remove_compress_chunks_policy(hypertable REGCLASS, if_exists BOOL = false) RETURNS BOOL
Expand Down
4 changes: 0 additions & 4 deletions sql/updates/1.2.2--1.3.0.sql
View file
Open in desktop
Expand Up @@ -103,7 +103,3 @@ ADD CONSTRAINT valid_job_type CHECK (job_type IN ('telemetry_and_version_check_i
ALTER TABLE _timescaledb_config.bgw_policy_drop_chunks
ADD COLUMN cascade_to_materializations BOOLEAN;
DROP FUNCTION IF EXISTS add_drop_chunks_policy(REGCLASS, INTERVAL, BOOL, BOOL);
CREATE OR REPLACE FUNCTION add_drop_chunks_policy(hypertable REGCLASS, older_than INTERVAL, cascade BOOL = FALSE, if_not_exists BOOL = false, cascade_to_materializations BOOL = false)
RETURNS INTEGER
AS '@MODULE_PATHNAME@', 'ts_add_drop_chunks_policy'
LANGUAGE C VOLATILE STRICT;
1 change: 1 addition & 0 deletions sql/updates/latest-dev.sql
View file
Open in desktop
Expand Up @@ -11,6 +11,7 @@ DROP FUNCTION _timescaledb_internal.partitioning_column_to_pretty;
-- Add new function definitions, columns and tables for distributed hypertables
DROP FUNCTION IF EXISTS create_hypertable(regclass,name,name,integer,name,name,anyelement,boolean,boolean,regproc,boolean,text,regproc,regproc);
DROP FUNCTION IF EXISTS add_drop_chunks_policy;
DROP FUNCTION IF EXISTS remove_drop_chunks_policy;
DROP FUNCTION IF EXISTS drop_chunks;

ALTER TABLE _timescaledb_catalog.hypertable ADD COLUMN replication_factor SMALLINT NULL CHECK (replication_factor > 0);
Expand Down
8 changes: 4 additions & 4 deletions src/cross_module_fn.c
View file
Open in desktop
Expand Up @@ -22,13 +22,13 @@

/* bgw policy functions */
CROSSMODULE_WRAPPER(add_compress_chunks_policy);
CROSSMODULE_WRAPPER(add_drop_chunks_policy);
CROSSMODULE_WRAPPER(add_retention_policy);
CROSSMODULE_WRAPPER(policy_reorder_add);
CROSSMODULE_WRAPPER(policy_reorder_proc);
CROSSMODULE_WRAPPER(policy_reorder_remove);
CROSSMODULE_WRAPPER(alter_job_schedule);
CROSSMODULE_WRAPPER(remove_compress_chunks_policy);
CROSSMODULE_WRAPPER(remove_drop_chunks_policy);
CROSSMODULE_WRAPPER(remove_retention_policy);

CROSSMODULE_WRAPPER(reorder_chunk);
CROSSMODULE_WRAPPER(move_chunk);
Expand Down Expand Up @@ -338,14 +338,14 @@ TSDLLEXPORT CrossModuleFunctions ts_cm_functions_default = {

/* bgw policies */
.add_compress_chunks_policy = error_no_default_fn_pg_community,
.add_drop_chunks_policy = error_no_default_fn_pg_community,
.add_retention_policy = error_no_default_fn_pg_community,
.policy_reorder_add = error_no_default_fn_pg_community,
.policy_reorder_proc = error_no_default_fn_pg_community,
.policy_reorder_remove = error_no_default_fn_pg_community,
.alter_job_schedule = error_no_default_fn_pg_community,
.bgw_policy_job_execute = bgw_policy_job_execute_default_fn,
.remove_compress_chunks_policy = error_no_default_fn_pg_community,
.remove_drop_chunks_policy = error_no_default_fn_pg_community,
.remove_retention_policy = error_no_default_fn_pg_community,

.move_chunk = error_no_default_fn_pg_enterprise,
.reorder_chunk = error_no_default_fn_pg_community,
Expand Down
4 changes: 2 additions & 2 deletions src/cross_module_fn.h
View file
Open in desktop
Expand Up @@ -48,12 +48,12 @@ typedef struct CrossModuleFunctions
bool (*bgw_policy_job_execute)(BgwJob *job);
bool (*continuous_agg_materialize)(int32 materialization_id, ContinuousAggMatOptions *options);

PGFunction add_drop_chunks_policy;
PGFunction add_retention_policy;
PGFunction policy_reorder_add;
PGFunction policy_reorder_proc;
PGFunction policy_reorder_remove;
PGFunction add_compress_chunks_policy;
PGFunction remove_drop_chunks_policy;
PGFunction remove_retention_policy;
PGFunction remove_compress_chunks_policy;

void (*create_upper_paths_hook)(PlannerInfo *, UpperRelationKind, RelOptInfo *, RelOptInfo *,
Expand Down
4 changes: 2 additions & 2 deletions test/expected/extension.out
View file
Open in desktop
Expand Up @@ -22,8 +22,8 @@ WHERE oid IN (
add_compress_chunks_policy
add_data_node
add_dimension
add_drop_chunks_policy
add_reorder_policy
add_retention_policy
allow_new_chunks
alter_job_schedule
attach_data_node
Expand Down Expand Up @@ -55,8 +55,8 @@ WHERE oid IN (
move_chunk
refresh_continuous_aggregate
remove_compress_chunks_policy
remove_drop_chunks_policy
remove_reorder_policy
remove_retention_policy
reorder_chunk
set_adaptive_chunking
set_chunk_time_interval
Expand Down
8 changes: 4 additions & 4 deletions tsl/src/bgw_policy/drop_chunks_api.c
View file
Open in desktop
Expand Up @@ -79,8 +79,8 @@ validate_drop_chunks_hypertable(Cache *hcache, Oid user_htoid, Oid older_than_ty
older_than = ts_interval_from_sql_input(meta->ht_oid,
older_than_datum,
older_than_type,
"older_than",
"add_drop_chunks_policy");
"retention_window",
"add_retention_policy");
}
else
{
Expand Down Expand Up @@ -109,8 +109,8 @@ validate_drop_chunks_hypertable(Cache *hcache, Oid user_htoid, Oid older_than_ty
older_than = ts_interval_from_sql_input_internal(open_dim,
older_than_datum,
older_than_type,
"older_than",
"add_drop_chunks_policy");
"retention_window",
"add_retention_policy");
}
Assert(meta->ht != NULL);
meta->older_than = older_than;
Expand Down
4 changes: 2 additions & 2 deletions tsl/src/init.c
View file
Open in desktop
Expand Up @@ -92,12 +92,12 @@ CrossModuleFunctions tsl_cm_functions = {
.add_tsl_telemetry_info = tsl_telemetry_add_info,
.bgw_policy_job_execute = tsl_bgw_policy_job_execute,
.continuous_agg_materialize = continuous_agg_materialize,
.add_drop_chunks_policy = drop_chunks_add_policy,
.add_retention_policy = drop_chunks_add_policy,
.policy_reorder_add = policy_reorder_add,
.policy_reorder_proc = policy_reorder_proc,
.policy_reorder_remove = policy_reorder_remove,
.add_compress_chunks_policy = compress_chunks_add_policy,
.remove_drop_chunks_policy = drop_chunks_remove_policy,
.remove_retention_policy = drop_chunks_remove_policy,
.remove_compress_chunks_policy = compress_chunks_remove_policy,
.create_upper_paths_hook = tsl_create_upper_paths_hook,
.set_rel_pathlist_dml = tsl_set_rel_pathlist_dml,
Expand Down
26 changes: 13 additions & 13 deletions tsl/test/expected/bgw_policy.out
View file
Open in desktop
Expand Up @@ -204,7 +204,7 @@ select remove_reorder_policy('test_table');
(1 row)

-- Now do drop_chunks test
select add_drop_chunks_policy('test_table', INTERVAL '4 months', true) as drop_chunks_job_id \gset
select add_retention_policy('test_table', INTERVAL '4 months', true) as drop_chunks_job_id \gset
SELECT count(*) FROM _timescaledb_catalog.chunk as c, _timescaledb_catalog.hypertable as ht where c.hypertable_id = ht.id and ht.table_name='test_table';
count
-------
Expand Down Expand Up @@ -258,9 +258,9 @@ select :before_count=:after_count;
t
(1 row)

select remove_drop_chunks_policy('test_table');
remove_drop_chunks_policy
---------------------------
select remove_retention_policy('test_table');
remove_retention_policy
-------------------------

(1 row)

Expand Down Expand Up @@ -436,7 +436,7 @@ WARNING: unexpected interval: smaller than one second

\set ON_ERROR_STOP 0
-- we cannot add a drop_chunks policy on a table whose open dimension is not time and no now_func is set
select add_drop_chunks_policy('test_table_int', INTERVAL '4 months', true);
select add_retention_policy('test_table_int', INTERVAL '4 months', true);
ERROR: integer_now_func not set on hypertable "test_table_int"
\set ON_ERROR_STOP 1
INSERT INTO test_table_int VALUES (-2, -2), (-1, -1), (0,0), (1, 1), (2, 2), (3, 3);
Expand Down Expand Up @@ -474,7 +474,7 @@ SELECT count(*) FROM _timescaledb_catalog.chunk as c, _timescaledb_catalog.hyper
6
(1 row)

select add_drop_chunks_policy('test_table_int', 1, true) as drop_chunks_job_id \gset
select add_retention_policy('test_table_int', 1, true) as drop_chunks_job_id \gset
-- Now simulate drop_chunks running automatically by calling it explicitly
select test_drop_chunks(:drop_chunks_job_id);
test_drop_chunks
Expand Down Expand Up @@ -627,7 +627,7 @@ select set_integer_now_func('test_overflow_smallint', 'overflow_now');

(1 row)

select add_drop_chunks_policy('test_overflow_smallint', -2) as drop_chunks_job_id \gset
select add_retention_policy('test_overflow_smallint', -2) as drop_chunks_job_id \gset
\set ON_ERROR_STOP 0
select test_drop_chunks(:drop_chunks_job_id);
ERROR: ts_interval overflow
Expand Down Expand Up @@ -682,7 +682,7 @@ select set_integer_now_func('part_time_now_func', 'dummy_now');

(1 row)

select add_drop_chunks_policy('part_time_now_func', 0) as drop_chunks_job_id \gset
select add_retention_policy('part_time_now_func', 0) as drop_chunks_job_id \gset
select test_drop_chunks(:drop_chunks_job_id);
test_drop_chunks
------------------
Expand All @@ -696,9 +696,9 @@ select * from part_time_now_func;
3.3 | 42.3
(2 rows)

select remove_drop_chunks_policy('part_time_now_func');
remove_drop_chunks_policy
---------------------------
select remove_retention_policy('part_time_now_func');
remove_retention_policy
-------------------------

(1 row)

Expand All @@ -723,9 +723,9 @@ select add_reorder_policy('test_table_perm', 'test_table_perm_pkey');
ERROR: must be owner of hypertable "test_table_perm"
select remove_reorder_policy('test_table');
ERROR: must be owner of hypertable "test_table"
select add_drop_chunks_policy('test_table_perm', INTERVAL '4 months', true);
select add_retention_policy('test_table_perm', INTERVAL '4 months', true);
ERROR: must be owner of hypertable "test_table_perm"
select remove_drop_chunks_policy('test_table');
select remove_retention_policy('test_table');
ERROR: must be owner of hypertable "test_table"
\set ON_ERROR_STOP 1
\c :TEST_DBNAME :ROLE_DEFAULT_PERM_USER
2 changes: 1 addition & 1 deletion tsl/test/expected/bgw_reorder_drop_chunks.out
View file
Open in desktop
Expand Up @@ -406,7 +406,7 @@ SELECT json_object_field(get_telemetry_report(always_display_report := true)::js
"0"
(1 row)

SELECT add_drop_chunks_policy('test_drop_chunks_table', INTERVAL '4 months') as drop_chunks_job_id \gset
SELECT add_retention_policy('test_drop_chunks_table', INTERVAL '4 months') as drop_chunks_job_id \gset
SELECT json_object_field(get_telemetry_report(always_display_report := true)::json,'num_drop_chunks_policies');
json_object_field
-------------------
Expand Down
2 changes: 1 addition & 1 deletion tsl/test/expected/compress_bgw_reorder_drop_chunks.out
View file
Open in desktop
Expand Up @@ -73,7 +73,7 @@ SELECT json_object_field(get_telemetry_report(always_display_report := true)::js
"0"
(1 row)

SELECT add_drop_chunks_policy('test_drop_chunks_table', INTERVAL '4 months') as drop_chunks_job_id \gset
SELECT add_retention_policy('test_drop_chunks_table', INTERVAL '4 months') as drop_chunks_job_id \gset
SELECT json_object_field(get_telemetry_report(always_display_report := true)::json,'num_drop_chunks_policies');
json_object_field
-------------------
Expand Down
2 changes: 1 addition & 1 deletion tsl/test/expected/compression_errors.out
View file
Open in desktop
Expand Up @@ -249,7 +249,7 @@ SELECT comp_hyper.schema_name|| '.' || comp_hyper.table_name as "COMPRESSED_HYPE
FROM _timescaledb_catalog.hypertable comp_hyper
INNER JOIN _timescaledb_catalog.hypertable uncomp_hyper ON (comp_hyper.id = uncomp_hyper.compressed_hypertable_id)
WHERE uncomp_hyper.table_name like 'foo' ORDER BY comp_hyper.id LIMIT 1 \gset
select add_drop_chunks_policy(:'COMPRESSED_HYPER_NAME', INTERVAL '4 months', true);
select add_retention_policy(:'COMPRESSED_HYPER_NAME', INTERVAL '4 months', true);
ERROR: cannot add drop chunks policy to compressed hypertable "_compressed_hypertable_14"
--Constraint checking for compression
create table fortable(col integer primary key);
Expand Down
16 changes: 8 additions & 8 deletions tsl/test/expected/continuous_aggs_bgw_drop_chunks.out
View file
Open in desktop
Expand Up @@ -69,7 +69,7 @@ REFRESH MATERIALIZED VIEW drop_chunks_view1;
--TEST1 specify drop chunks policy on mat. hypertable by
-- directly does not work
\set ON_ERROR_STOP 0
SELECT add_drop_chunks_policy( '_timescaledb_internal._materialized_hypertable_2', older_than=> -50, cascade_to_materializations=>false ) as drop_chunks_job_id1 \gset
SELECT add_retention_policy( '_timescaledb_internal._materialized_hypertable_2', retention_window=> -50, cascade_to_materializations=>false ) as drop_chunks_job_id1 \gset
ERROR: cannot add drop chunks policy to materialized hypertable "_materialized_hypertable_2"
\set ON_ERROR_STOP 1
--TEST2 specify drop chunks policy on cont. aggregate
Expand All @@ -94,7 +94,7 @@ ORDER BY ranges;
_timescaledb_internal._hyper_2_43_chunk | {"[30,40)"}
(4 rows)

SELECT add_drop_chunks_policy( 'drop_chunks_view1', older_than=> 10, cascade_to_materializations=>false ) as drop_chunks_job_id1 \gset
SELECT add_retention_policy( 'drop_chunks_view1', retention_window=> 10, cascade_to_materializations=>false ) as drop_chunks_job_id1 \gset
SELECT alter_job_schedule(:drop_chunks_job_id1, schedule_interval => INTERVAL '1 second');
alter_job_schedule
-----------------------------------------------------
Expand All @@ -113,15 +113,15 @@ SELECT count(c) from show_chunks('_timescaledb_internal._materialized_hypertable
1
(1 row)

SELECT remove_drop_chunks_policy('drop_chunks_view1');
remove_drop_chunks_policy
---------------------------
SELECT remove_retention_policy('drop_chunks_view1');
remove_retention_policy
-------------------------

(1 row)

\set ON_ERROR_STOP 0
SELECT remove_drop_chunks_policy('unknown');
ERROR: relation "unknown" does not exist at character 34
SELECT remove_drop_chunks_policy('1');
SELECT remove_retention_policy('unknown');
ERROR: relation "unknown" does not exist at character 32
SELECT remove_retention_policy('1');
ERROR: OID 1 does not refer to a hypertable or continuous aggregate
\set ON_ERROR_STOP 1