Problems when a table grows beyond 65535 pointer pages [CORE4384]

[firebird-automations]

Submitted by: @ibprovider

Attachments:
miscelan-logs.zip
inserts-batch-and-sql.zip

The USHORT pp_sequence number wraps and it may lead to unexpected issues like an infinite loop inside extend_relation() or locate_space(). This issue manifests itself when a table (including garbage) gets bigger than 256GB (for 4KB page) ... 4TB (for 16KB page) and remains even after a garbage collection pass.

Commits: 1655ba4 bd3ce4e FirebirdSQL/fbt-repository@ec2c79f FirebirdSQL/fbt-repository@3d5e270

====== Test Details ======

There is no .fbt file for this ticket, but I've done such test and result was: table CAN be of such size that more than 64K PP will be related to it.

[firebird-automations]

Modified by: @dyemanov

reporter: Dmitry Yemanov [ dimitr ] => Kovalenko Dmitry [ _dima_k_ ]

assignee: Dmitry Yemanov [ dimitr ]

[firebird-automations]

Modified by: @dyemanov

Fix Version: 3.0 Beta 2 [ 10586 ]

[firebird-automations]

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.4 [ 10585 ]

[firebird-automations]

Commented by: @pavel-zotov

http://svn.code.sf.net/p/firebird/code/firebird/trunk/ChangeLog:
> 2014-12-29 17:40 dimitr <...>
> This commit is expected to fix CORE4384: Problems when a table grows beyond 65535 pointer pages. Testing would be appreciated.

I've done somewhat like test about ability to create and fill table with more than 65K pointer pages.
Short conclusion: we CAN create such table, at least on database with page_size = 4K.

Environment: OS = Windows XP, FB = WI-T6.3.0.31543, SuperServer; database = page_4k, fw_OFF

firebird.conf:

DefaultDbCachePages = 64K
LockHashSlots = 22111
TempCacheLimit = 2147483647

RemoteServicePort = 3333

MaxUnflushedWrites = -1
MaxUnflushedWriteTime = -1

ExternalFileAccess = Restrict C:\1INSTALL\FBTEST
MaxUserTraceLogSize = 99999

FileSystemCacheThreshold = 65536K
AuthServer = Legacy_Auth,Srp
AuthClient = Legacy_Auth,Srp,Win_Sspi
WireCrypt = Disabled
UserManager = Legacy_UserManager
BugcheckAbort = 1

Two tables are in this database:

-- main:
recreate table t(id bigint, s01 varchar(8), s02 varchar(8));

-- log with info for later analyzing of inserts speed:
recreate table ins_log(
dts_beg timestamp, dts_end timestamp,
elapsed_ms int computed by(datediff(millisecond from dts_beg to dts_end)),
gen_after_ins bigint, pp_after_ins int
);
commit;

recreate sequence g;
commit;

Test was run during several days, final database size = ~305Gb.

Result:

Database "E:\DATA\HUGE_CORE_4384.FDB"
Database header page information:
Flags 0
Generation 14032
System Change Number 0
Page size 4096
ODS version 12.0
Oldest transaction 1149
Oldest active 11496
Oldest snapshot 11496
Next transaction 11497
Sequence number 0
Next attachment ID 5741
Implementation HW=Intel/i386 little-endian OS=Windows CC=MSVC
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Creation date Jan 18, 2015 19:02:14
Attributes

Variable header data:
\*END\*

Database file sequence:
File E:\DATA\HUGE_CORE_4384.FDB is the only file

Analyzing database pages ...
T (128)
Primary pointer page: 211, Index root page: 212
Pointer pages: 127855, data page slots: 103306800
Data pages: 103306800, average fill: 67%
Primary pages: 103306800, secondary pages: 0, swept pages: 0
Empty pages: 5, full pages: 103306794
Fill distribution:
0 - 19% = 5
20 - 39% = 0
40 - 59% = 1
60 - 79% = 103306794
80 - 99% = 0

I decided to check trivial COUNT operation on this table and create INDEX on it.

Results are in attach "miscelan-logs.zip".

[firebird-automations]

Modified by: @pavel-zotov

Attachment: miscelan-logs.zip [ 12670 ]

Attachment: inserts-batch-and-sql.zip [ 12671 ]

[firebird-automations]

Modified by: @pavel-zotov

Attachment: fill_huge_table_log_until_count_3e9_fw-ON.zip [ 12674 ]

[firebird-automations]

Modified by: @dyemanov

Attachment: fill_huge_table_log_until_count_3e9_fw-ON.zip [ 12674 ] =>

[firebird-automations]

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: Done successfully

Test Details: There is no .fbt file for this ticket, but I've done such test and result was: table CAN be of such size that more than 64K PP will be related to it.

[firebird-automations]

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]