[Bug]: Unable to drop extension after compressing a table in extension schema. #4140

JamesGuthrie · 2022-03-03T13:05:58Z

What type of bug is this?

Unexpected error

What subsystems and features are affected?

Compression

What happened?

When creating and then compressing a table during extension installation, it is no longer possible to drop the extension.

TimescaleDB version affected

2.5.2

PostgreSQL version used

14

What operating system did you use?

Ubuntu, MacOS

What installation method did you use?

Docker, Source, Other

What platform did you run on?

Other

Relevant log output and stack trace

test=# CREATE EXTENSION nouninstall;
CREATE EXTENSION
test=# DROP EXTENSION nouninstall;
ERROR:  cache lookup failed for relation 0

How can we reproduce the bug?

Place the following .control and .sql file in pg's extension directory, then in a psql shell execute CREATE EXTENSION nouninstall; DROP EXTENSION nouninstall;

// nouninstall.control
comment = 'Timescale nouninstaller'
default_version = '0.1'
module_pathname = '$libdir/nouninstall'
relocatable = false
schema = nouninstall
superuser = true

// nouninstall--0.1.sql
CREATE SCHEMA IF NOT EXISTS _ps_trace;
CREATE SCHEMA IF NOT EXISTS ps_trace;

CREATE DOMAIN ps_trace.trace_id uuid NOT NULL CHECK (value != '00000000-0000-0000-0000-000000000000');

CREATE TABLE IF NOT EXISTS _ps_trace.span
(
    trace_id ps_trace.trace_id NOT NULL,
    span_id bigint NOT NULL,
    parent_span_id bigint NULL,
    start_time timestamptz NOT NULL,
    end_time timestamptz NOT NULL,
    PRIMARY KEY (span_id, trace_id, start_time),
    CHECK (start_time <= end_time)
);
CREATE INDEX ON _ps_trace.span USING BTREE (trace_id, parent_span_id) INCLUDE (span_id); -- used for recursive CTEs for trace tree queries

SELECT public.create_hypertable(
        '_ps_trace.span'::regclass,
        'start_time'::name,
        partitioning_column=>'trace_id'::name,
        number_partitions=>1::int,
        chunk_time_interval=>'07:57:57.345608'::interval,
        create_default_indexes=>false
    );

ALTER TABLE _ps_trace.span SET (timescaledb.compress, timescaledb.compress_segmentby='trace_id,span_id');

The text was updated successfully, but these errors were encountered:

mkindahl · 2022-03-03T13:35:08Z

Quick reproduction with some additional information of where the error is generated:

mats@fury:~/issues/timescaledb-4140$ psql
Expanded display is used automatically.
Null display is "[NULL]".
psql (13.5)
Type "help" for help.

mats=# create extension nouninstall;
CREATE EXTENSION
mats=# drop extension nouninstall ;
ERROR:  cache lookup failed for relation 0
mats=# \set VERBOSITY verbose
mats=# drop extension nouninstall ;
ERROR:  XX000: cache lookup failed for relation 0
LOCATION:  getRelationIdentity, objectaddress.c:5283
mats=# \dx
                                       List of installed extensions
    Name     |  Version  |   Schema    |                            Description                            
-------------+-----------+-------------+-------------------------------------------------------------------
 nouninstall | 0.1       | nouninstall | Timescale nouninstaller
 plpgsql     | 1.0       | pg_catalog  | PL/pgSQL procedural language
 timescaledb | 2.6.0-dev | public      | Enables scalable inserts and complex queries for time-series data
(3 rows)

JamesGuthrie · 2022-03-03T13:41:03Z

In case it's helpful, here's a backtrace of a breakpoint set at cache lookup failed for relation 0 (timescale 2.5.2, pg 14.2):

#0  getRelationIdentity (buffer=buffer@entry=0x7ffdb157dcb0, relid=0,
    object=object@entry=0x55bcbff457a8, missing_ok=missing_ok@entry=false)
    at objectaddress.c:5860
#1  0x000055bcbe3b6eab in getObjectIdentityParts (
    object=object@entry=0x55bcbff45778, objname=objname@entry=0x55bcbff457a8,
    objargs=objargs@entry=0x55bcbff457b0, missing_ok=missing_ok@entry=false)
    at objectaddress.c:4697
#2  0x000055bcbe43bd19 in EventTriggerSQLDropAddObject (
    object=object@entry=0x55bcc02d0120, original=original@entry=true,
    normal=normal@entry=false) at event_trigger.c:1269
#3  0x000055bcbe39f30a in deleteObjectsInList (
    targetObjects=targetObjects@entry=0x55bcc0333cd8,
    depRel=depRel@entry=0x7ffdb157ddd8, flags=flags@entry=0) at dependency.c:250
#4  0x000055bcbe39f3e0 in performDeletion (object=object@entry=0x7ffdb157de24,
    behavior=behavior@entry=DROP_RESTRICT, flags=flags@entry=0) at dependency.c:353
#5  0x00007f8fab78be57 in ts_hypertable_drop (hypertable=0x55bcbf5525f0,
    behavior=behavior@entry=DROP_RESTRICT)
    at /home/james/timescaledb/src/hypertable.c:775
#6  0x00007f8fab78c4b4 in hypertable_tuple_delete (ti=0x7ffdb157dea0,
    data=<optimized out>) at /home/james/timescaledb/src/hypertable.c:705
#7  0x00007f8fab7a12e9 in ts_scanner_scan (ctx=ctx@entry=0x7ffdb157df30)
    at /home/james/timescaledb/src/scanner.c:306
#8  0x00007f8fab7899db in hypertable_scan_limit_internal (
    scankey=scankey@entry=0x7ffdb157e000, num_scankeys=num_scankeys@entry=2,
    indexid=indexid@entry=1,
    on_tuple_found=on_tuple_found@entry=0x7f8fab78c3a5 <hypertable_tuple_delete>,
    scandata=scandata@entry=0x0, limit=limit@entry=0, lock=3, tuplock=false,
    mctx=0x55bcbf356c90, filter=0x0) at /home/james/timescaledb/src/hypertable.c:383
#9  0x00007f8fab78bdbc in ts_hypertable_delete_by_name (
    schema_name=0x55bcc032b820 "_ps_trace", table_name=<optimized out>)
    at /home/james/timescaledb/src/hypertable.c:731
#10 0x00007f8fab79b21e in process_drop_table (obj=0x55bcc032b8c0)
    at /home/james/timescaledb/src/process_utility.c:4096
#11 0x00007f8fab79b460 in process_ddl_sql_drop (obj=<optimized out>)
    at /home/james/timescaledb/src/process_utility.c:4176
#12 0x00007f8fab79b4ce in process_ddl_event_sql_drop (
    trigdata=trigdata@entry=0x7ffdb157e300)
    at /home/james/timescaledb/src/process_utility.c:4304
#13 0x00007f8fab7a0d01 in ts_timescaledb_process_ddl_event (fcinfo=0x7ffdb157e140)
    at /home/james/timescaledb/src/process_utility.c:4327
#14 0x000055bcbe43afd6 in EventTriggerInvoke (
    fn_oid_list=fn_oid_list@entry=0x55bcbfc873b8,
    trigdata=trigdata@entry=0x7ffdb157e300) at event_trigger.c:920
#15 0x000055bcbe43b99c in EventTriggerSQLDrop (
    parsetree=parsetree@entry=0x55bcbeeed3d0) at event_trigger.c:791
#16 0x000055bcbe680e25 in ProcessUtilitySlow (pstate=pstate@entry=0x55bcbf743788,
    pstmt=pstmt@entry=0x55bcbeeedd50,
    queryString=queryString@entry=0x55bcbeeec930 "DROP EXTENSION promscale;",
    context=context@entry=PROCESS_UTILITY_TOPLEVEL, params=params@entry=0x0,
    queryEnv=queryEnv@entry=0x0, dest=0x55bcbeeedde8, qc=0x7ffdb157ea00)
    at utility.c:1906
#17 0x000055bcbe67fc40 in standard_ProcessUtility (pstmt=0x55bcbeeedd50,
    queryString=0x55bcbeeec930 "DROP EXTENSION promscale;",
    readOnlyTree=<optimized out>, context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
    queryEnv=0x0, dest=0x55bcbeeedde8, qc=0x7ffdb157ea00) at utility.c:978
#18 0x00007f8fb46c70e0 in loader_process_utility_hook (pstmt=0x55bcbeeedd50,
    query_string=0x55bcbeeec930 "DROP EXTENSION promscale;",
    readonly_tree=<optimized out>, context=PROCESS_UTILITY_TOPLEVEL, params=0x0,
    queryEnv=0x0, dest=0x55bcbeeedde8, completion_tag=0x7ffdb157ea00)
    at /home/james/timescaledb/src/loader/loader.c:563
#19 0x00007f8fab79a4c4 in prev_ProcessUtility (args=args@entry=0x7ffdb157e6c0)
    at /home/james/timescaledb/src/process_utility.c:90
#20 0x00007f8fab7a050a in timescaledb_ddl_command_start (pstmt=0x55bcbeeedd50,
    query_string=<optimized out>, readonly_tree=<optimized out>,
    context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=<optimized out>,
    dest=0x55bcbeeedde8, completion_tag=0x7ffdb157ea00)
    at /home/james/timescaledb/src/process_utility.c:4264
#21 0x000055bcbe67fe51 in ProcessUtility (pstmt=pstmt@entry=0x55bcbeeedd50,
    queryString=<optimized out>, readOnlyTree=<optimized out>,
    context=<optimized out>, params=<optimized out>, queryEnv=<optimized out>,
    dest=0x55bcbeeedde8, qc=0x7ffdb157ea00) at utility.c:523
#22 0x000055bcbe67d3ee in PortalRunUtility (portal=portal@entry=0x55bcbef54c40,
    pstmt=pstmt@entry=0x55bcbeeedd50, isTopLevel=isTopLevel@entry=true,
    setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x55bcbeeedde8,
    @entry=0x7ffdb157ea00) at pquery.c:1155
#23 0x000055bcbe67d5e3 in PortalRunMulti (portal=portal@entry=0x55bcbef54c40, isTopLevel=isTopLevel@entry=true, setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x55bcbeeedde8, altdest=altdest@entry=0x55bcbeeedde8, qc=qc@entry=0x7ffdb157ea00)
    at pquery.c:1312
#24 0x000055bcbe67dafd in PortalRun (portal=portal@entry=0x55bcbef54c40, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0x55bcbeeedde8, altdest=altdest@entry=0x55bcbeeedde8,
    qc=0x7ffdb157ea00) at pquery.c:788
#25 0x000055bcbe679c36 in exec_simple_query (query_string=query_string@entry=0x55bcbeeec930 "DROP EXTENSION promscale;") at postgres.c:1214
#26 0x000055bcbe67bc81 in PostgresMain (argc=argc@entry=1, argv=argv@entry=0x7ffdb157ebe0, dbname=<optimized out>, username=<optimized out>) at postgres.c:4486
#27 0x000055bcbe5da94c in BackendRun (port=port@entry=0x55bcbef12740) at postmaster.c:4530
#28 0x000055bcbe5ddbae in BackendStartup (port=port@entry=0x55bcbef12740) at postmaster.c:4252
#29 0x000055bcbe5dddc9 in ServerLoop () at postmaster.c:1745
#30 0x000055bcbe5df2f8 in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at postmaster.c:1417
#31 0x000055bcbe520220 in main (argc=3, argv=0x55bcbeee6570) at main.c:209

mkindahl · 2022-03-03T14:02:01Z

At this point in hypertable_tuple_delete the function ts_hypertable_get_by_id returns a hypertable with the main_table_relid set to zero, which then is used to try to locate a table.

	if (!compressed_hypertable_id_isnull)
	{
		Hypertable *compressed_hypertable = ts_hypertable_get_by_id(compressed_hypertable_id);
		/* The hypertable may have already been deleted by a cascade */
		if (compressed_hypertable != NULL)
			ts_hypertable_drop(compressed_hypertable, DROP_RESTRICT);
	}

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. Fixes timescale#4140

mkindahl · 2022-03-04T13:52:34Z

The bug occurs because when deleting the extension nouninstall it will trigger a lookup of all dependent objects by looking into pg_depend to find all objects that are dependent on the extension. In this particular case, you would get this result:

mats=# select classid::regclass, case classid when 'pg_class'::regclass then objid::regclass::text when 'pg_namespace'::regclass then objid::regnamespace::text when 'pg_type'::regclass then objid::regtype::text end as object from pg_depend where refclassid = 'pg_extension'::regclass and refobjid = 203438;
   classid    |                     object                      
--------------+-------------------------------------------------
 pg_namespace | _ps_trace
 pg_namespace | ps_trace
 pg_type      | ps_trace.trace_id
 pg_class     | _ps_trace.span
 pg_class     | _timescaledb_internal._compressed_hypertable_10
(5 rows)

These tables will now be deleted and the passed to the sql_drop trigger and we will attempt to delete them one by one. Since the tables are deleted first, the _timescaledb_internal._compressed_hypertable_10 does not exist when we run the sql_drop trigger on _ps_trace.span. It is still in the metadata, but it does not exist in the catalog any more, hence triggering an error because get_relname_relid() returns 0 on a non-existing relation.

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. This scenario can occur if both the hypertable and a compressed table are deleted as part of a dependency, for example, if a compressed hypertable is defined inside an extension. In this case, the compressed hypertable will be deleted first, and the lookup of the compressed hypertable will find it in the metadata but a lookup of the actual table will fail. Fixes timescale#4140

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. This scenario can occur if both the hypertable and a compressed table are deleted as part of running a `sql_drop` event, for example, if a compressed hypertable is defined inside an extension. In this case, the compressed hypertable (indeed all tables) will be deleted first, and the lookup of the compressed hypertable will find it in the metadata but a lookup of the actual table will fail since the table does not exist. Fixes timescale#4140

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. This scenario can occur if both the hypertable and a compressed table are deleted as part of running a `sql_drop` event, for example, if a compressed hypertable is defined inside an extension. In this case, the compressed hypertable (indeed all tables) will be deleted first, and the lookup of the compressed hypertable will find it in the metadata but a lookup of the actual table will fail since the table does not exist. Fixes #4140

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. This scenario can occur if both the hypertable and a compressed table are deleted as part of running a `sql_drop` event, for example, if a compressed hypertable is defined inside an extension. In this case, the compressed hypertable (indeed all tables) will be deleted first, and the lookup of the compressed hypertable will find it in the metadata but a lookup of the actual table will fail since the table does not exist. Fixes timescale#4140

When running `performDeletion` is is necessary to have a valid relation id, but when doing a lookup using `ts_hypertable_get_by_id` this might actually return a hypertable entry pointing to a table that does not exist because it has been deleted previously. In this case, only the catalog entry should be removed, but it is not necessary to delete the actual table. This scenario can occur if both the hypertable and a compressed table are deleted as part of running a `sql_drop` event, for example, if a compressed hypertable is defined inside an extension. In this case, the compressed hypertable (indeed all tables) will be deleted first, and the lookup of the compressed hypertable will find it in the metadata but a lookup of the actual table will fail since the table does not exist. Fixes #4140

JamesGuthrie added the bug label Mar 3, 2022

svenklemm added the compression label Mar 3, 2022

mkindahl mentioned this issue Mar 3, 2022

Ignore invalid relid when deleting hypertable #4142

Merged

mkindahl self-assigned this Mar 3, 2022

JamesGuthrie mentioned this issue Mar 7, 2022

Drop Extension is Broken timescale/promscale_extension#95

Closed

mkindahl closed this as completed in #4142 Mar 14, 2022

mkindahl added this to the TimescaleDB 2.6.1 milestone Mar 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Unable to drop extension after compressing a table in extension schema. #4140

[Bug]: Unable to drop extension after compressing a table in extension schema. #4140

JamesGuthrie commented Mar 3, 2022 •

edited

mkindahl commented Mar 3, 2022

JamesGuthrie commented Mar 3, 2022 •

edited

mkindahl commented Mar 3, 2022

mkindahl commented Mar 4, 2022 •

edited

[Bug]: Unable to drop extension after compressing a table in extension schema. #4140

[Bug]: Unable to drop extension after compressing a table in extension schema. #4140

Comments

JamesGuthrie commented Mar 3, 2022 • edited

What type of bug is this?

What subsystems and features are affected?

What happened?

TimescaleDB version affected

PostgreSQL version used

What operating system did you use?

What installation method did you use?

What platform did you run on?

Relevant log output and stack trace

How can we reproduce the bug?

mkindahl commented Mar 3, 2022

JamesGuthrie commented Mar 3, 2022 • edited

mkindahl commented Mar 3, 2022

mkindahl commented Mar 4, 2022 • edited

JamesGuthrie commented Mar 3, 2022 •

edited

JamesGuthrie commented Mar 3, 2022 •

edited

mkindahl commented Mar 4, 2022 •

edited