[Bug]: Transaction aborted incorrectly if "out of memory" error occurs when inserting into hypertable

[alexanderlaw]

What type of bug is this?

Crash

What subsystems and features are affected?

Query executor

What happened?

When INSERT INTO hypertable can not be executed due to the error "out of shared memory", it triggers an assertion failure.

TimescaleDB version affected

2.12.0-dev

PostgreSQL version used

15.3

What operating system did you use?

Ubuntu 22.04 x86_64

What installation method did you use?

Source

What platform did you run on?

On prem/Self-hosted

Relevant log output and stack trace

INSERT INTO t SELECT generate_series('0001-01-01'::timestamp, '2023-08-01'::timestamp, '1 day');
ERROR:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.
WARNING:  AbortTransaction while in ABORT state
ERROR:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
connection to server was lost

Core was generated by `postgres: law regression [local] INSERT                                       '.
Program terminated with signal SIGABRT, Aborted.

warning: Section `.reg-xstate/1314830' in core file too small.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140585902083456) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140585902083456) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140585902083456) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140585902083456, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fdcb424d476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fdcb42337f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x000055f3ac42b397 in ExceptionalCondition (
    conditionName=0x55f3ac6041c0 "TransactionIdIsValid(proc->xid)", 
    errorType=0x55f3ac603fb4 "FailedAssertion", fileName=0x55f3ac603fa8 "procarray.c", lineNumber=682)
    at assert.c:69
#6  0x000055f3ac225ab9 in ProcArrayEndTransaction (proc=0x7fdcb3eb1650, latestXid=728) at procarray.c:682
#7  0x000055f3abdee021 in AbortTransaction () at xact.c:2823
#8  0x000055f3abdeea5b in AbortCurrentTransaction () at xact.c:3327
#9  0x000055f3ac2617da in PostgresMain (dbname=0x55f3ad469458 "regression", username=0x55f3ad469438 "law")
    at postgres.c:4322
#10 0x000055f3ac186f8d in BackendRun (port=0x55f3ad460b70) at postmaster.c:4511
#11 0x000055f3ac186814 in BackendStartup (port=0x55f3ad460b70) at postmaster.c:4239
#12 0x000055f3ac1826e7 in ServerLoop () at postmaster.c:1806
#13 0x000055f3ac181e44 in PostmasterMain (argc=3, argv=0x55f3ad3a2a70) at postmaster.c:1478
#14 0x000055f3ac075ad5 in main (argc=3, argv=0x55f3ad3a2a70) at main.c:202

How can we reproduce the bug?

CREATE EXTENSION timescaledb;

CREATE TABLE t(time timestamptz NOT NULL);
SELECT create_hypertable('t', 'time');

INSERT INTO t SELECT generate_series('0001-01-01'::timestamp, '2023-08-01'::timestamp, '1 day');

[konskov]

Thank you for the bug report @alexanderlaw and for providing detailed steps to reproduce, I was able to reproduce it.

[nikkhils]

@alexanderlaw does this happen only in DEBUG builds for you?

[alexanderlaw]

@nikkhils , yes. I've tried to reproduce the issue with timescaledb built with:

$ ./bootstrap -DCMAKE_BUILD_TYPE=Release ...
$ cd ./build && make && make install
...
-- Install configuration: "Release"
...

And I see no assertion failure:

INSERT INTO t SELECT generate_series('0001-01-01'::timestamp, '2023-08-01'::timestamp, '1 day');
ERROR:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

(Postgres was built with "./configure --enable-debug --enable-cassert").

[nikkhils]

thanks for the confirmation that it only happens with Debug builds. I am working on a fix for this 👍

Btw, the out of shared memory occurs because of a large number of chunks creation since we are inserting a very large time range worth of data in one transaction and the corresponding locking required for them.