Skip to content

Stuff That Should Be Written Down Somewhere

m42a edited this page Apr 5, 2012 · 6 revisions

For technical reasons, Lirch doesn't close its modules when they've quit. This does not, however, introduce a memory leak. The reason for this is that modules aren't written to. Therefore, a module that has been loaded multiple times can share code with its other instances; the only memory specific to that module is the memory allocated for running the module's thread, and that gets freed when the thread exits. Therefore, loading and unloading a module multiple times only causes a O(1) memory leak rather than a O(n) memory leak. However, since the module hasn't been written to it's mmaped instead of copied into RAM, and thus the memory for it can be deallocated under memory pressure and simply be read from disk if the need arises, so although loading and unloading many different modules will increase the program's virtual size, it will have no lasting effect on the program's resident memory size, which is the measurement that actually matters.

So we have numbers now. With all of the main plugins loaded and the basic UI, Lirch takes up 560MB of virtual memory. However, only 4MB of that memory is actually committed, and 3.2MB of that is shared (which means multiple processes can use it without having multiple copies in memory). The upshot of this is that despite the fact that we're leaking virtual memory like a wet paper sieve, we only take up 4MB of actual RAM and each additional invocation of the program costs only 800KB. And since we have 18EB of virtual memory, we're not going to run out any time soon, so this isn't really a problem.

The technical reason that Lirch can't unload modules when it's done with them is that modules are allowed to define their own message types. Every message type derives from message_data, which requires it to have both a virtual destructor and a virtual copy function. If a module was unloaded and a message that was defined in that module was still circulating around other plugins, any attempt to copy or destroy that message would, if we're lucky, call into unmapped memory, which will immediately segfault our program. If we're unlucky, another module has been loaded into that memory, and we're calling directly into arbitrary code, which is really really really bad.

Lirch is fully Unicode compliant. All data transmitted over the network is encoded in UTF-8, and all data presented to the user is done in accordance with the user's locale settings. Internal data is generally encoded as UTF-16. Both UTF-8 and UTF-16 are variable-length encodings; a UTF-8 character can be anywhere from 1 to 4 bytes and a UTF-16 character is either 2 or 4 bytes. This means that you can't split strings at arbitrary points, since you might be splitting up a character. However, both UTF-8 and UTF-16 represent characters in such a way that no character's encoding is a subsequence of another character's encoding, which means if you use a find function to look for a particular character sequence it's okay to split the string at the beginning or end of that sequence, since you know that no characters will be split if you do that. For example the character U+1F438 (🐸) consists of 2 UTF-16 chars: U+D83D and U+DC38. If you split those up, the character will not display properly. In addition, please try to not to store Unicode strings in std::strings, since those are not Unicode aware, and might not handle character encoding properly.

A message's type is determined by a string, so if 2 modules define different messages with the same mangled name, the compiler will think they're the same type. Don't do that.

The QSettings Documentation is a dirty liar; having 2 threads construct separate QSettings objects will not work (except for sometimes, when it does, but these occurrences appear to be random). It will, however, appear to work until the program attempts to exit, at which point the program will attempt to free memory that is not mapped, and so will dump core. Unfortunately, there's no way to fix this that doesn't involve a wrapper class, a smart pointer, and a mutex. So get on that, someone who isn't me. It's important to note that this is not code we can fix; the crash happens in the compiler-generated code that is called after main exits, and there's no portable way to change that code. The best we can do is pretend that it's not there and try really hard not to touch it.

Clone this wiki locally