"Operating system call pthread_mutex_destroy failed. Error code 16" in firebird.log, server crashed

[fperana]

A problem similar to that discussed in #7371 appeared.

I'm running a patched version of FB 4.0.2 x64 on Linux Debian 11. The patch is a fix for the aforementioned issue.

Core dump follows:

core_20230321T171237.001.zip
core_20230321T171237.002.zip
core_20230321T171237.003.zip

The server crashed immediately after this error was logged in firebird.log.

Originally posted by @fperana in #7371 (comment)

[AlexPeshkoff]

Are you using replication?

[fperana]

No, not in Firebird. It is implemented at system level (the whole VM is replicated by VMWare).

[AlexPeshkoff]

Unfortunately the only one thing left to do now is to wait for next dump and hope it's more informative.

The big problem with debugging MT is that state of problematic object even after segfault may be changed by other threads cause process does not stop atomically. In current dump state of problematic mutex is OK, not busy like it was at crash moment. I.e. I can not know what thread was locking it at that time. Moreover, all threads are in inactive state and I can suppose one that caused an issue might even finish already.

[fperana]

New event today, same error. Here's the coredump.

core_20230531T112951.001.zip
core_20230531T112951.002.zip
core_20230531T112951.003.zip
core_20230531T112951.004.zip

[fperana]

Another one...

core_20230608T162138.001.zip
core_20230608T162138.002.zip
core_20230608T162138.003.zip

[AlexPeshkoff]

I need your binaries & debug info to do something, I did not keep them that long.

[fperana]

Here they are.

https://drive.google.com/file/d/1mMoXIhjHOLm0vMhmYNLAA7QjefR4TEP6/view?usp=sharing
https://drive.google.com/file/d/1orJyWvhuuRtemsnZKhQZH5AYDhUq_PeE/view?usp=sharing

[AlexPeshkoff]

In all cases repeat same problem - problematic mutex is OK in the dump. What's worse is that mutexes are different, i.e. almost any mutex may be bad from you host POV - but it's OK in the dump. I can only suggest to upgrade linux, or check HW, or may be check VM soft if you run on VM - I doubt what we get here is FB problem.