Message "Operating system call munmap failed. Error code 12" can appear in firebird.log under heavy load (2.5.x, CS, SC) [CORE5643]

[firebird-automations]

Submitted by: @pavel-zotov

Alex asked me to open special ticket for this issue.

Host has several databases which are served by single FB instance. Total number of attachments in work time is ~550.
After some time of normal work following message "Operating system call munmap failed. Error code 12" - did appear in firebird.log four times, and after that fb_smp_server was crashed.
I've sent stack-trace and additional info by e-mail, 18.10.2017 23:48.

Commits: d9f53fa f5a37fd a65276b

[firebird-automations]

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

[firebird-automations]

Modified by: @AlexPeshkoff

summary: Message "Operating system call munmap failed. Error code 12" can apper in firebird.log under heavy load (2.5.x, CS, SC) => Message "Operating system call munmap failed. Error code 12" can appear in firebird.log under heavy load (2.5.x, CS, SC)

[firebird-automations]

Commented by: @AlexPeshkoff

Certainly one can't change Linux behavior regarding splitting VMA (Virtual Memory Area) structures in munmap() call.
What we get with this fix - memory block which can't be unmapped due to ENOMEM error is saved in linked list in MemoryPool and reused as soon as possible. I.e. we avoid throwing exception (which often happens in dtors and therefore highly unliked by c++11) and do not leak memory on this error. Remaining problem is unmapping such memory when firebird dynamic library (i.e. plugin) is unloaded - cleanup code does it's best to unmap linked blocks but if it fails (VMAs allocated by our library are mixed with allocated by others) memory leak appears unavoidable.

Solution was tested artificially reserving most of VMAs using code from https://stackoverflow.com/questions/43743555/munmap-failure-with-enomem-with-private-anonymous-mapping. After applying this fix SC server 2.5 stopped to fail randomly during massive deallocation of memory (was caused stopping TPC-C) and demonstrated stable behavior without memory leaks.

[firebird-automations]

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.8 [ 10809 ]

Fix Version: 3.0.3 [ 10810 ]

Fix Version: 4.0 Beta 1 [ 10750 ]

[firebird-automations]

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Cannot be tested

Test Specifics: [Platform (Windows/Linux) specific]

[firebird-automations]

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]