SMP support improvements #6

wants to merge 8 commits into


None yet
2 participants

rraptorr commented Sep 11, 2011

Recently I stumbled upon a problem with exmpp_tls driver. I was pushing lots of data through it, when I realized that I am limited to only one processor core. After some digging in the code I've made some changes to improve SMP support in exmpp drivers (not only TLS). This branch covers following topics, further details can be found in commit messages and code comments:

  • Port level locking in OpenSSL TLS driver.
    This is kinda quirky since OTP already links with OpenSSL, so I've tried to make use of that. This was the least error prone way I could think of (fighting with crypto module over OpenSSL callbacks could have caused quite o mayhem).
  • SMP support in stringprep.
    Port level locking and using multiple ports as shown in Erlang Efficiency Guide.
  • Port level locking in zlib driver.
    This one is straight forward.
  • Some memory leaks I've found in OpenSSL driver
  • I've also enabled extended marker with versioning in all drivers.
    Versioning is a nice feature and has nothing to do with SMP and is supported by R12B5 (which I believe is the lowest Erlang version exmpp supports) so I've enabled it unconditionally.

All of the changes were tested on R12B5, R13B04 and R14B03. I've written simple modules to launch multiple stringprep/tls/zlib processes to verify that I can utilize as much processor cores as possible. Results were best on R14B03 (I've utilized all four cores on my machine), but also R13B04 and R12B5 benefit from those changes.

Looking forward to any comments about this.

rraptorr added some commits Sep 10, 2011

Enable port level locking in stringprep driver.
By default Erlang uses driver level locking.
However, it seems there is no reason for stringprep driver not to
use port level locking, thus reducing lock contention.
Enable port level locking for OpenSSL TLS driver.
OTP crypto module is already using OpenSSL and sets some
global OpenSSL callbacks that are needed to make it thread-safe.
To avoid problems with own callbacks, start crypto before
loading OpenSSL driver and enable port level locking if the
needed callbacks are already set.
Eliminate lock contention on stringprep port
exmpp_stringprep module used to open only one port, so there
was still a lot of lock contention on port access.
Now it opens as many ports as there are scheduler threads
(as described in Erlang Efficiency Guide).
Use ERTS allocation functions in exmpp_compress_zlib
exmpp_compress_zlib now configures zlib to use ERTS
allocation functions (driver_alloc/driver_free)
which are guaranteed to be thread-safe.
Enable port level locking in zlib compression driver.
zlib is thread-safe (as long as memory allocation functions it
uses are thread safe) so enable port level locking in zlib
compression driver.
Fix memory leaks in OpenSSL TLS driver.
SSL context and connection structures were never freed.
Additionally, trusted_certs also was never freed.
Use extended marker and driver versioning without SMP support.
Extended marker indicates that driver supports additional flags
like versioning and port level locking. Currently it was only
used when SMP support was enabled. Versioning seems like a nice
feature and has nothing to do with SMP (and is present in R12B5)
so enable it regardless of SMP support.
Fix compatibility with R12B5
There is no min/2 function in R12B5.

badlop commented Sep 12, 2011

Thanks. I've briefly tested, and gave a quick review. Pushed to master.

@badlop badlop closed this Sep 12, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment