Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

SMP support improvements #6

Closed
wants to merge 8 commits into from

2 participants

@rraptorr

Recently I stumbled upon a problem with exmpp_tls driver. I was pushing lots of data through it, when I realized that I am limited to only one processor core. After some digging in the code I've made some changes to improve SMP support in exmpp drivers (not only TLS). This branch covers following topics, further details can be found in commit messages and code comments:

  • Port level locking in OpenSSL TLS driver. This is kinda quirky since OTP already links with OpenSSL, so I've tried to make use of that. This was the least error prone way I could think of (fighting with crypto module over OpenSSL callbacks could have caused quite o mayhem).
  • SMP support in stringprep. Port level locking and using multiple ports as shown in Erlang Efficiency Guide.
  • Port level locking in zlib driver. This one is straight forward.
  • Some memory leaks I've found in OpenSSL driver
  • I've also enabled extended marker with versioning in all drivers. Versioning is a nice feature and has nothing to do with SMP and is supported by R12B5 (which I believe is the lowest Erlang version exmpp supports) so I've enabled it unconditionally.

All of the changes were tested on R12B5, R13B04 and R14B03. I've written simple modules to launch multiple stringprep/tls/zlib processes to verify that I can utilize as much processor cores as possible. Results were best on R14B03 (I've utilized all four cores on my machine), but also R13B04 and R12B5 benefit from those changes.

Looking forward to any comments about this.

rraptorr added some commits
@rraptorr rraptorr Enable port level locking in stringprep driver.
By default Erlang uses driver level locking.
However, it seems there is no reason for stringprep driver not to
use port level locking, thus reducing lock contention.
3c18f1a
@rraptorr rraptorr Enable port level locking for OpenSSL TLS driver.
OTP crypto module is already using OpenSSL and sets some
global OpenSSL callbacks that are needed to make it thread-safe.
To avoid problems with own callbacks, start crypto before
loading OpenSSL driver and enable port level locking if the
needed callbacks are already set.
e0596e6
@rraptorr rraptorr Eliminate lock contention on stringprep port
exmpp_stringprep module used to open only one port, so there
was still a lot of lock contention on port access.
Now it opens as many ports as there are scheduler threads
(as described in Erlang Efficiency Guide).
071900e
@rraptorr rraptorr Use ERTS allocation functions in exmpp_compress_zlib
exmpp_compress_zlib now configures zlib to use ERTS
allocation functions (driver_alloc/driver_free)
which are guaranteed to be thread-safe.
a335a3d
@rraptorr rraptorr Enable port level locking in zlib compression driver.
zlib is thread-safe (as long as memory allocation functions it
uses are thread safe) so enable port level locking in zlib
compression driver.
b0a55e1
@rraptorr rraptorr Fix memory leaks in OpenSSL TLS driver.
SSL context and connection structures were never freed.
Additionally, trusted_certs also was never freed.
2ae9af3
@rraptorr rraptorr Use extended marker and driver versioning without SMP support.
Extended marker indicates that driver supports additional flags
like versioning and port level locking. Currently it was only
used when SMP support was enabled. Versioning seems like a nice
feature and has nothing to do with SMP (and is present in R12B5)
so enable it regardless of SMP support.
1594df2
@rraptorr rraptorr Fix compatibility with R12B5
There is no min/2 function in R12B5.
046ddc4
@badlop
Collaborator

Thanks. I've briefly tested, and gave a quick review. Pushed to master.

@badlop badlop closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 12, 2011
  1. @rraptorr

    Enable port level locking in stringprep driver.

    rraptorr authored
    By default Erlang uses driver level locking.
    However, it seems there is no reason for stringprep driver not to
    use port level locking, thus reducing lock contention.
  2. @rraptorr

    Enable port level locking for OpenSSL TLS driver.

    rraptorr authored
    OTP crypto module is already using OpenSSL and sets some
    global OpenSSL callbacks that are needed to make it thread-safe.
    To avoid problems with own callbacks, start crypto before
    loading OpenSSL driver and enable port level locking if the
    needed callbacks are already set.
  3. @rraptorr

    Eliminate lock contention on stringprep port

    rraptorr authored
    exmpp_stringprep module used to open only one port, so there
    was still a lot of lock contention on port access.
    Now it opens as many ports as there are scheduler threads
    (as described in Erlang Efficiency Guide).
  4. @rraptorr

    Use ERTS allocation functions in exmpp_compress_zlib

    rraptorr authored
    exmpp_compress_zlib now configures zlib to use ERTS
    allocation functions (driver_alloc/driver_free)
    which are guaranteed to be thread-safe.
  5. @rraptorr

    Enable port level locking in zlib compression driver.

    rraptorr authored
    zlib is thread-safe (as long as memory allocation functions it
    uses are thread safe) so enable port level locking in zlib
    compression driver.
  6. @rraptorr

    Fix memory leaks in OpenSSL TLS driver.

    rraptorr authored
    SSL context and connection structures were never freed.
    Additionally, trusted_certs also was never freed.
  7. @rraptorr

    Use extended marker and driver versioning without SMP support.

    rraptorr authored
    Extended marker indicates that driver supports additional flags
    like versioning and port level locking. Currently it was only
    used when SMP support was enabled. Versioning seems like a nice
    feature and has nothing to do with SMP (and is present in R12B5)
    so enable it regardless of SMP support.
  8. @rraptorr

    Fix compatibility with R12B5

    rraptorr authored
    There is no min/2 function in R12B5.
This page is out of date. Refresh to see the latest.
View
23 c_src/exmpp_compress_zlib.c
@@ -41,6 +41,18 @@ struct exmpp_compress_zlib_data {
(to_send)->index); \
exmpp_free_xbuf((to_send));
+static void*
+exmpp_compress_zlib_alloc(void *opaque, unsigned int items, unsigned int size)
+{
+ return driver_alloc(items*size);
+}
+
+static void
+exmpp_compress_zlib_free(void *opaque, void *addr)
+{
+ driver_free(addr);
+}
+
/* -------------------------------------------------------------------
* Erlang port driver callbacks.
* ------------------------------------------------------------------- */
@@ -63,8 +75,8 @@ exmpp_compress_zlib_start(ErlDrvPort port, char *command)
edd->inf_z.next_in = edd->def_z.next_in = 0;
edd->inf_z.avail_in = edd->def_z.avail_in = 0;
- edd->inf_z.zalloc = edd->def_z.zalloc = NULL;
- edd->inf_z.zfree = edd->def_z.zfree = NULL;
+ edd->inf_z.zalloc = edd->def_z.zalloc = exmpp_compress_zlib_alloc;
+ edd->inf_z.zfree = edd->def_z.zfree = exmpp_compress_zlib_free;
edd->inf_z.opaque = edd->def_z.opaque = NULL;
return (ErlDrvData)edd;
@@ -336,6 +348,11 @@ static ErlDrvEntry compress_zlib_driver_entry = {
DRIVER_INIT(DRIVER_NAME)
{
-
+ compress_zlib_driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
+ compress_zlib_driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
+ compress_zlib_driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
+ compress_zlib_driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
+#endif
return &compress_zlib_driver_entry;
}
View
8 c_src/exmpp_stringprep.c
@@ -394,9 +394,15 @@ static ErlDrvEntry driver_entry;
DRIVER_INIT(DRIVER_NAME)
{
-
driver_entry.driver_name = S(DRIVER_NAME);
driver_entry.control = exmpp_stringprep_control;
+ driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
+ driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
+ driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
+ driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
+#endif
+
return (&driver_entry);
}
View
31 c_src/exmpp_tls_openssl.c
@@ -112,6 +112,12 @@ exmpp_tls_openssl_stop(ErlDrvData drv_data)
driver_free(edd->private_key);
if (edd->expected_id != NULL)
driver_free(edd->expected_id);
+ if (edd->trusted_certs != NULL)
+ driver_free(edd->trusted_certs);
+ if (edd->ssl != NULL)
+ SSL_free(edd->ssl);
+ if (edd->ctx != NULL)
+ SSL_CTX_free(edd->ctx);
driver_free(edd);
}
@@ -839,7 +845,6 @@ static ErlDrvEntry tls_openssl_driver_entry = {
DRIVER_INIT(DRIVER_NAME)
{
-
/* Initialize OpenSSL. */
SSL_library_init();
SSL_load_error_strings();
@@ -854,5 +859,29 @@ DRIVER_INIT(DRIVER_NAME)
ssl_ex_index = SSL_get_ex_new_index(0, "exmpp_tls_openssl_data",
NULL, NULL, NULL);
+ tls_openssl_driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
+ tls_openssl_driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
+ tls_openssl_driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
+ /**
+ * To make OpenSSL thread-safe, two callbacks must be set
+ * as described in http://www.openssl.org/docs/crypto/threads.html
+ *
+ * However, OTP comes with crypto module, that links with OpenSSL
+ * and sets the needed callbacks itself. If another set of
+ * callbacks had been provided here, it would overwrite
+ * or be overwritten by those from crypto module.
+ *
+ * So instead of providing callbacks, start crypto module
+ * from Erlang code before loading this driver. As a result
+ * crypto module will install the needed callbacks and
+ * this driver also can be made thread safe.
+ */
+ if (CRYPTO_get_locking_callback() != NULL &&
+ CRYPTO_get_id_callback() != NULL) {
+ tls_openssl_driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
+ }
+#endif
+
return &tls_openssl_driver_entry;
}
View
3  c_src/exmpp_xml_expat.c
@@ -515,7 +515,6 @@ static ErlDrvEntry driver_entry;
DRIVER_INIT(DRIVER_NAME)
{
-
driver_entry.driver_name = S(DRIVER_NAME);
driver_entry.init = exmpp_xml_init;
driver_entry.finish = exmpp_xml_finish;
@@ -523,10 +522,10 @@ DRIVER_INIT(DRIVER_NAME)
driver_entry.stop = exmpp_xml_stop;
driver_entry.control = exmpp_xml_control;
-#if defined(SMP_SUPPORT)
driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
#endif
View
3  c_src/exmpp_xml_expat_legacy.c
@@ -401,7 +401,6 @@ static ErlDrvEntry driver_entry;
DRIVER_INIT(DRIVER_NAME)
{
-
driver_entry.driver_name = S(DRIVER_NAME);
driver_entry.init = exmpp_xml_init;
driver_entry.finish = exmpp_xml_finish;
@@ -409,10 +408,10 @@ DRIVER_INIT(DRIVER_NAME)
driver_entry.stop = exmpp_xml_stop;
driver_entry.control = exmpp_xml_control;
-#if defined(SMP_SUPPORT)
driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
#endif
View
3  c_src/exmpp_xml_libxml2.c
@@ -460,7 +460,6 @@ static ErlDrvEntry driver_entry;
DRIVER_INIT(DRIVER_NAME)
{
-
driver_entry.driver_name = S(DRIVER_NAME);
driver_entry.init = exmpp_xml_init;
driver_entry.finish = exmpp_xml_finish;
@@ -468,10 +467,10 @@ DRIVER_INIT(DRIVER_NAME)
driver_entry.stop = exmpp_xml_stop;
driver_entry.control = exmpp_xml_control;
-#if defined(SMP_SUPPORT)
driver_entry.extended_marker = ERL_DRV_EXTENDED_MARKER;
driver_entry.major_version = ERL_DRV_EXTENDED_MAJOR_VERSION;
driver_entry.minor_version = ERL_DRV_EXTENDED_MINOR_VERSION;
+#if defined(SMP_SUPPORT)
driver_entry.driver_flags = ERL_DRV_FLAG_USE_PORT_LOCKING;
#endif
View
57 src/core/exmpp_stringprep.erl
@@ -57,11 +57,28 @@
code_change/3
]).
--record(state, {port}).
+-record(state, {ports}).
-define(SERVER, ?MODULE).
-define(DRIVER_NAME, exmpp_stringprep).
--define(PORT_REGISTERED_NAME, exmpp_stringprep_port).
+
+%% http://www.erlang.org/doc/efficiency_guide/drivers.html#id68009
+-define(PORT_REGISTERED_NAMES, {exmpp_stringprep_port01,
+ exmpp_stringprep_port02,
+ exmpp_stringprep_port03,
+ exmpp_stringprep_port04,
+ exmpp_stringprep_port05,
+ exmpp_stringprep_port06,
+ exmpp_stringprep_port07,
+ exmpp_stringprep_port08,
+ exmpp_stringprep_port09,
+ exmpp_stringprep_port10,
+ exmpp_stringprep_port11,
+ exmpp_stringprep_port12,
+ exmpp_stringprep_port13,
+ exmpp_stringprep_port14,
+ exmpp_stringprep_port15,
+ exmpp_stringprep_port16}).
-define(COMMAND_LOWERCASE, 0).
-define(COMMAND_NAMEPREP, 1).
@@ -249,15 +266,16 @@ port_revision() ->
string() | {error, invalid_string | exmpp_not_started}.
control(Command, String) ->
+ PortName = port_name(erlang:system_info(scheduler_id)),
try
- case port_control(?PORT_REGISTERED_NAME, Command, String) of
+ case port_control(PortName, Command, String) of
[0 | _] -> {error, invalid_string};
[1 | Result] -> Result
end
catch
error:badarg ->
- case erlang:port_info(?PORT_REGISTERED_NAME, registered_name) of
- {registered_name, ?PORT_REGISTERED_NAME} ->
+ case erlang:port_info(PortName, registered_name) of
+ {registered_name, PortName} ->
{error, invalid_string};
undefined ->
{error, exmpp_not_started}
@@ -290,6 +308,11 @@ control_reuse_arg(Command, String) ->
Other
end.
+-spec port_name(pos_integer()) -> atom().
+port_name(N) ->
+ element(N rem tuple_size(?PORT_REGISTERED_NAMES) + 1,
+ ?PORT_REGISTERED_NAMES).
+
%% --------------------------------------------------------------------
%% gen_server(3erl) callbacks.
%% --------------------------------------------------------------------
@@ -299,10 +322,22 @@ control_reuse_arg(Command, String) ->
init([]) ->
try
exmpp_internals:load_driver(?DRIVER_NAME),
- Port = exmpp_internals:open_port(?DRIVER_NAME),
- register(?PORT_REGISTERED_NAME, Port),
+ Schedulers = erlang:system_info(schedulers),
+ PortCount =
+ if Schedulers > tuple_size(?PORT_REGISTERED_NAMES) ->
+ tuple_size(?PORT_REGISTERED_NAMES);
+ true ->
+ Schedulers
+ end,
+ Ports =
+ lists:map(fun(N) ->
+ Port = exmpp_internals:open_port(?DRIVER_NAME),
+ register(port_name(N), Port),
+ Port
+ end,
+ lists:seq(1, PortCount)),
State = #state{
- port = Port
+ ports = Ports
},
{ok, State}
catch
@@ -343,7 +378,9 @@ code_change(Old_Vsn, State, Extra) ->
%% @hidden
-terminate(_Reason, #state{port = Port} = _State) ->
- exmpp_internals:close_port(Port),
+terminate(_Reason, #state{ports = Ports} = _State) ->
+ lists:foreach(fun(Port) ->
+ exmpp_internals:close_port(Port)
+ end, Ports),
exmpp_internals:unload_driver(?DRIVER_NAME),
ok.
View
5 src/core/exmpp_tls.erl
@@ -139,6 +139,11 @@ start_link() ->
-ifdef(HAVE_OPENSSL).
-define(REGISTER_OPENSSL,
+ %% crypto module installs various global OpenSSL callbacks
+ %% that make OpenSSL thread-safe. The OpenSSL driver will
+ %% detect and make use of it, so ensure that crypto is
+ %% started before loading the driver.
+ crypto:start(),
register_builtin_engine(openssl, exmpp_tls_openssl,
[{x509, 10}])).
-else.
Something went wrong with that request. Please try again.