Firebird 2.5.1 runs out of memory while restoring database backup [CORE3802]

[firebird-automations]

Submitted by: Artem Petkevych (artem)

Is related to CORE3433

While restoring table with array fields, Firebird consumes all available memory and restore fails with error "unable to allocate memory from operating system". Same backup can be successfully restored by Firebird 2.1.4. DDL of the table is:

CREATE TABLE RVG (
RYEAR SMALLINT NOT NULL,
INS_ID BIGINT NOT NULL,
S1 DECIMAL(18,2),
S16 DECIMAL(18,2),
R1 DECIMAL(18,2) [1:12],
R4 DECIMAL(18,2) [1:4],
R5 DECIMAL(18,2) [1:4],
R6 DECIMAL(18,2) [1:4],
VK1 COMPUTED BY (R1[1]+R1[2]+R1[3]),
VK2 COMPUTED BY (R1[4]+R1[5]+R1[6]),
VK3 COMPUTED BY (R1[7]+R1[8]+R1[9]),
VK4 COMPUTED BY (R1[10]+R1[11]+R1[12]),
VG COMPUTED BY (R1[1]+R1[2]+R1[3]+R1[4]+R1[5]+R1[6]+R1[7]+R1[8]+R1[9]+R1[10]+R1[11]+R1[12]),
R3 INTEGER [1:4]
);

ALTER TABLE RVG ADD CONSTRAINT PK_RVG PRIMARY KEY (RYEAR, INS_ID);

Table has 5454042 rows. Error occurs after 2560000 records restored.
To reproduce issue one can download backup from this URL: http://www.protec.kiev.ua/~artem/db/rvg.7z

Commits: b153879 c6c1c2c 373ec81

[firebird-automations]

Commented by: @dyemanov

It can be reproduced with 2.5.1 as well, hence I've changed the subject to mention the already released version instead of a snapshot build.

[firebird-automations]

Modified by: @dyemanov

Version: 2.5.1 [ 10333 ]

environment: Windows 7 64-bit, RAM 4 gb, Firebird 2.5.2.26449 32-bit => Windows 7 64-bit, RAM 4 gb, Firebird 2.5.1.26351 32-bit

summary: Firebird 2.5.2 runs out of memory while restoring database backup => Firebird 2.5.1 runs out of memory while restoring database backup

[firebird-automations]

Modified by: @dyemanov

Link: This issue is related to CORE3433 [ CORE3433 ]

[firebird-automations]

Commented by: @dyemanov

This is a regression introduced while implementing CORE3433.

[firebird-automations]

Commented by: @dyemanov

Usage of isc_send() instead of isc_start_and_send() forces the looper to always exit with a non-NULL node, so release_blobs() is never called and thus blobs/arrays bound to the current request are never released, at least until the transaction ends. Hence we have a transaction level memory leak.

The dumb but easiest solution could be to restart the request periodically (e.g. if (records % 1000 == 1)) to keep some kind of a balance between memory usage and performance. Or set some flag if we have blobs/arrays stored right before this record and use that flag to restart the request instead of just sending the record. Or combine these two approaches.

Ideally, the engine should be fixed, but this sounds too risky for point releases.

[firebird-automations]

Modified by: @dyemanov

assignee: Adriano dos Santos Fernandes [ asfernandes ]

[firebird-automations]

Commented by: @asfernandes

That's weird. I tested latest 2.5 in release-mode, with and without CORE3433 change.

Around 2,400,000 records I verified and in both cases top says virtual memory consumption is around 370M.

I can restore it in Linux 64 bits.

Without CORE3433, it takes "3m6.838s" and with it "2m53.671s".

[firebird-automations]

Commented by: @dyemanov

With a 32-bit FB 2.5.2 (release mode) on 64-bit Windows I see VM = 750MB after 1,000,000 records restored, 1.5GB after 2,000,000 records and the OOM error happens at 2,262,000 records. 64-bit FB consumes a bit more memory but unsurprisingly succeeds to restore.

Just for the record, I was restoring using the Services API.

[firebird-automations]

Modified by: @asfernandes

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.2 [ 10450 ]

[firebird-automations]

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

[firebird-automations]

Modified by: @pavel-zotov

status: Closed [ 6 ] => Closed [ 6 ]

QA Status: No test