fsserver: corrupted double-linked list

[wehimwich]

@jfhquick reported that after 110 days of running continuously, fsserver crashed with a "corrupted double-linked list" error:

So it is FS 10.2.0 running on FSL8 (32-bit). The trigger seems to have been an open client where the ssh timed out due to the remote end going away. This after many months and many tens of client connections.

The fsserver error log is atttached: fsserver.2024.Aug.20.13.00.36.err.log

[dehorsley]

Interesting. The "corrupted double-linked list" message looks to be coming from malloc, indicating some memory corruption somewhere. It doesn't look like fsserver was compiled with symbols (-g) so we can't be sure if it's in the server code or one of the libraries (jansson or nng) but I'd guess in fsserver code. A quick look on their GitHub issue trackers for the projects doesn't show up anything obvious. I'll see if I can dust off my VM and Valgrind.

[wehimwich]

Thanks for this. You make a case for having -g as the default.

@jfhquick may be able to monitor the memory usage of long running instances of fsserver.

I having no experience with valgrind myself, but ran it with:

valgrind --leak-check=full --log-file=/tmp/valgrind.log --show-leak-kinds=all -v bin/fsserver start

Then ran the FS, disconnected the cliient, attached a new one, and then terminated (which also terminated fsserver).

valgrind.log is attached. If it would save you steps, feel free to send suggestions.