Permalink
Browse files

initial support for dirty schedulers and dirty NIFs

Add initial support for dirty schedulers.

There are two types of dirty schedulers: CPU schedulers and I/O
schedulers. By default, there are as many dirty CPU schedulers as there are
normal schedulers and as many dirty CPU schedulers online as normal
schedulers online. There are 10 dirty I/O schedulers (similar to the choice
of 10 as the default for async threads).

By default, dirty schedulers are disabled and conditionally compiled
out. To enable them, you must pass --enable-dirty-schedulers to the
top-level configure script when building Erlang/OTP.

Current dirty scheduler support requires the emulator to be built with SMP
support. This restriction will be lifted in the future.

You can specify the number of dirty schedulers with the command-line
options +SDcpu (for dirty CPU schedulers) and +SDio (for dirty I/O
schedulers). The +SDcpu option is similar to the +S option in that it takes
two numbers separated by a colon: C1:C2, where C1 specifies the number of
dirty schedulers available and C2 specifies the number of dirty schedulers
online. The +SDPcpu option allows numbers of dirty CPU schedulers available
and dirty CPU schedulers online to be specified as percentages, similar to
the existing +SP option for normal schedulers. The number of dirty CPU
schedulers created and dirty CPU schedulers online may not exceed the
number of normal schedulers created and normal schedulers online,
respectively. The +SDio option takes only a single number specifying the
number of dirty I/O schedulers available and online. There is no support
yet for programmatically changing at run time the number of dirty CPU
schedulers online via erlang:system_flag/2. Also, changing the number of
normal schedulers online via erlang:system_flag(schedulers_online,
NewSchedulersOnline) should ensure that there are no more dirty CPU
schedulers than normal schedulers, but this is not yet implemented. You can
retrieve the number of dirty schedulers by passing dirty_cpu_schedulers,
dirty_cpu_schedulers_online, or dirty_io_schedulers to
erlang:system_info/1.

Currently only NIFs are able to access dirty scheduler
functionality. Neither drivers nor BIFs currently support dirty
schedulers. This restriction will be addressed in the future.

If dirty scheduler support is present in the runtime, the initial status
line Erlang prints before presenting its interactive prompt will include
the indicator "[ds:C1:C2:I]" where "ds" indicates "dirty schedulers", "C1"
indicates the number of dirty CPU schedulers available, "C2" indicates the
number of dirty CPU schedulers online, and "I" indicates the number of
dirty I/O schedulers.

Document The dirty NIF API in the erl_nif man page. The API closely follows
Rickard Green's presentation slides from his talk "Future Extensions to the
Native Interface", presented at the 2011 Erlang Factory held in the San
Francisco Bay Area. Rickard's slides are available online at
http://bit.ly/1m34UHB .

Document the new erl command-line options, the additions to
erlang:system_info/1, and also add the erlang:system_flag/2 dirty scheduler
documentation even though it's not yet implemented.

To determine whether the dirty NIF API is available, native code can check
to see whether the C preprocessor macro ERL_NIF_DIRTY_SCHEDULER_SUPPORT is
defined. To check if dirty schedulers are available at run time, native
code can call the boolean enif_have_dirty_schedulers() function, and Erlang
code can call erlang:system_info(dirty_cpu_schedulers), which raises
badarg if no dirty scheduler support is available.

Add a simple dirty NIF test to the emulator NIF suite.
  • Loading branch information...
vinoski authored and rickard-green committed Jan 10, 2014
1 parent aa1b402 commit c1c03ae4ee50e58b7669ea88ec4d29c6b2b67c7b
@@ -315,6 +315,11 @@ Some of the available `configure` options are:
no automatic dependency handling inbetween applications. So if you disable
an application that another depends on, you also have to disable the
dependant application.
* `--enable-dirty-schedulers` - Enable the **experimental** dirty schedulers
functionality. Note that the dirty schedulers functionality is experimental,
and **not supported**. This functionality **will** be subject to backward
incompatible changes. Note that you should **not** enable the dirty scheduler
functionality on production systems. It is only provided for testing.
If you or your system has special requirements please read the `Makefile` for
additional configuration information.
@@ -196,6 +196,9 @@ AC_ARG_ENABLE(threads,
AS_HELP_STRING([--enable-threads], [enable async thread support])
AS_HELP_STRING([--disable-threads], [disable async thread support]))
AC_ARG_ENABLE(dirty-schedulers,
AS_HELP_STRING([--enable-dirty-schedulers], [enable dirty scheduler support]))
AC_ARG_ENABLE(halfword-emulator,
AS_HELP_STRING([--enable-halfword-emulator],
[enable halfword emulator (only for 64bit builds)]))
@@ -101,7 +101,7 @@ ERL_XCOMP_SYSROOT_INIT
AC_ISC_POSIX
AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in)
AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in include/$host/erl_native_features_config.h:include/erl_native_features_config.h.in)
dnl ----------------------------------------------------------------------
dnl Optional features.
dnl ----------------------------------------------------------------------
@@ -123,6 +123,7 @@ AS_HELP_STRING([--enable-bootstrap-only],
with_ssl_zlib=no
enable_hipe=no
enable_sctp=no
enable_dirty_schedulers=no
fi
])
@@ -134,6 +135,13 @@ AS_HELP_STRING([--disable-threads], [disable async thread support]),
*) enable_threads=yes ;;
esac ], enable_threads=unknown)
AC_ARG_ENABLE(dirty-schedulers,
AS_HELP_STRING([--enable-dirty-schedulers], [enable dirty scheduler support]),
[ case "$enableval" in
no) enable_dirty_schedulers=no ;;
*) enable_dirty_schedulers=yes ;;
esac ], enable_dirty_schedulers=no)
AC_ARG_ENABLE(halfword-emulator,
AS_HELP_STRING([--enable-halfword-emulator],
[enable halfword emulator (only for 64bit builds)]),
@@ -1222,6 +1230,13 @@ esac
if test $emu_threads != yes; then
enable_lock_check=no
enable_lock_count=no
AC_MSG_CHECKING(whether dirty schedulers should be enabled)
if test "x$enable_dirty_schedulers" != "xno"; then
AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support])
AC_MSG_RESULT(yes)
else
AC_MSG_RESULT(no)
fi
else
# Threads enabled for emulator
EMU_THR_LIB_NAME=$ETHR_LIB_NAME
@@ -1240,7 +1255,17 @@ else
if test "x$enable_lock_count" != "xno"; then
EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_ENABLE_LOCK_COUNT"
fi
AC_MSG_CHECKING(whether dirty schedulers should be enabled)
if test "x$enable_dirty_schedulers" != "xno"; then
EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_DIRTY_SCHEDULERS"
AC_DEFINE(ERTS_DIRTY_SCHEDULERS, 1, [Define if the emulator supports dirty schedulers])
AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support])
AC_MSG_RESULT(yes)
else
AC_MSG_RESULT(no)
fi
disable_child_waiter_thread=no
case $host_os in
solaris*)
@@ -792,6 +792,54 @@
SMP support enabled (see the <seealso marker="#smp">-smp</seealso>
flag).</p>
</item>
<tag><marker id="+SDcpu"><c><![CDATA[+SDcpu DirtyCPUSchedulers:DirtyCPUSchedulersOnline]]></c></marker></tag>
<item>
<p>Sets the number of dirty CPU scheduler threads to create and dirty
CPU scheduler threads to set online when threading support has been
enabled. The maximum for both values is 1024, and each value is further
limited by the settings for normal schedulers: the number of dirty CPU
scheduler threads created cannot exceed the number of normal scheduler
threads created, and the number of dirty CPU scheduler threads online
cannot exceed the number of normal scheduler threads online (see the
<seealso marker="#+S">+S</seealso> and <seealso marker="#+SP">+SP</seealso>
flags for more details). By default, the number of dirty CPU scheduler
threads created equals the number of normal scheduler threads created,
and the number of dirty CPU scheduler threads online equals the number
of normal scheduler threads online. <c>DirtyCPUSchedulers</c> may be
omitted if <c>:DirtyCPUSchedulersOnline</c> is not and vice versa. The
number of dirty CPU schedulers online can be changed at run time via
<seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
</p>
<p>This option is ignored if the emulator doesn't have threading support
enabled. Currently, <em>this option is experimental</em> and is supported only
if the emulator was configured and built with support for dirty schedulers
enabled (it's disabled by default).
</p>
</item>
<tag><marker id="+SDPcpu"><c><![CDATA[+SDPcpu DirtyCPUSchedulersPercentage:DirtyCPUSchedulersOnlinePercentage]]></c></marker></tag>
<item>
<p>Similar to <seealso marker="#+SDcpu">+SDcpu</seealso> but uses percentages to set the
number of dirty CPU scheduler threads to create and number of dirty CPU scheduler threads
to set online when threading support has been enabled. Specified values must be greater
than 0. For example, <c>+SDPcpu 50:25</c> sets the number of dirty CPU scheduler threads
to 50% of the logical processors configured and the number of dirty CPU scheduler threads
online to 25% of the logical processors available. <c>DirtyCPUSchedulersPercentage</c> may
be omitted if <c>:DirtyCPUSchedulersOnlinePercentage</c> is not and vice versa. The
number of dirty CPU schedulers online can be changed at run time via
<seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
</p>
<p>This option interacts with <seealso marker="#+SDcpu">+SDcpu</seealso> settings.
For example, on a system with 8 logical cores configured and 8 logical cores available,
the combination of the options <c>+SDcpu 4:4 +SDPcpu 50:25</c> (in either order) results
in 2 dirty CPU scheduler threads (50% of 4) and 1 dirty CPU scheduler thread online (25% of 4).
</p>
<p>This option is ignored if the emulator doesn't have threading support
enabled. Currently, <em>this option is experimental</em> and is supported only
if the emulator was configured and built with support for dirty schedulers
enabled (it's disabled by default).
</p>
</item>
<tag><marker id="+SDio"><c><![CDATA[+SDio IOSchedulers]]></c></marker></tag>
<tag><c><![CDATA[+sFlag Value]]></c></tag>
<item>
<p>Scheduling specific flags.</p>
@@ -181,7 +181,11 @@ ok
to dispatch the work to another thread, return
from the native function, and wait for the result. The thread can send
the result back to the calling thread using message passing. Information
about thread primitives can be found below.</p>
about thread primitives can be found below. If you have built your system
with <em>the currently experimental</em> support for dirty schedulers,
you may want to try out this functionality by dispatching the work to a
<seealso marker="#dirty_nifs">dirty NIF</seealso>,
which does not have the same duration restriction as a normal NIF.</p>
</description>
<section>
<title>FUNCTIONALITY</title>
@@ -312,6 +316,38 @@ ok
<p>The library initialization callbacks <c>load</c>, <c>reload</c> and
<c>upgrade</c> are all thread-safe even for shared state data.</p>
</item>
<tag>Dirty NIFs</tag>
<item><p><marker id="dirty_nifs"/><em>Note that the dirty NIF functionality
is experimental</em> and that you have to enable support for dirty
schedulers when building OTP in order to try the functionality out. Native functions
<seealso marker="#lengthy_work">
must normally run quickly</seealso>, as explained earlier in this document. They
generally should execute for no more than a millisecond. But not all native functions
can execute so quickly; for example, functions that encrypt large blocks of data or
perform lengthy file system operations can often run for tens of seconds or more.</p>
<p>A NIF that cannot execute in a millisecond or less is called a "dirty NIF" since
it performs work that the Erlang runtime cannot handle cleanly. Applications
that make use of such functions must indicate to the runtime that the functions are
dirty so they can be handled specially. To schedule a dirty NIF for execution, the
application calls <seealso marker="#enif_schedule_dirty_nif">enif_schedule_dirty_nif</seealso>,
passing to it a pointer to the dirty NIF to be executed and indicating with a flag
argument whether it expects the operation to be CPU-bound or I/O-bound.</p>
<p>All dirty NIFs must ultimately invoke the <seealso marker="#enif_schedule_dirty_nif_finalizer">
enif_schedule_dirty_nif_finalizer</seealso> as their final action, passing to it the
result they wish to return to the original caller. A finalizer function can either
receive the result and return it directly, or it can return a different value instead.
For convenience, the NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
enif_dirty_nif_finalizer</seealso> function that applications can use as a finalizer;
it simply returns its result argument.</p>
<note><p>Dirty NIF support is available only when the emulator is configured with dirty
schedulers enabled. This feature is currently disabled by default. To determine whether
the dirty NIF API is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined. Also, if the Erlang runtime was built
without threading support, dirty schedulers are disabled. To check at runtime for the presence
of dirty scheduler threads, code can call the <seealso marker="#enif_have_dirty_schedulers"><c>
enif_have_dirty_schedulers()</c></seealso> API function, which returns true if dirty
scheduler threads are present, false otherwise.</p></note>
</item>
</taglist>
</section>
<section>
@@ -610,6 +646,18 @@ typedef enum {
See also the <seealso marker="#WARNING">warning</seealso> text at the beginning of this document.</p>
</desc>
</func>
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result)</nametext></name>
<fsummary>Simple dirty NIF result finalizer</fsummary>
<desc>
<p>A convenience function that a dirty NIF can use as a finalizer that simply
return its <c>result</c> argument as its return value. This function is provided
for dirty NIFs with results that should be returned directly to the original caller.</p>
<note><p>This function is available only when the emulator is configured with dirty
schedulers enabled. This feature is currently disabled by default. To determine whether
the dirty NIF API is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
</desc>
</func>
<func><name><ret>int</ret><nametext>enif_equal_tids(ErlNifTid tid1, ErlNifTid tid2)</nametext></name>
<fsummary></fsummary>
<desc><p>Same as <seealso marker="erl_driver#erl_drv_equal_tids">erl_drv_equal_tids</seealso>.
@@ -730,6 +778,22 @@ typedef enum {
and return true, or return false if <c>term</c> is not an unsigned integer or is
outside the bounds of type <c>unsigned long</c>.</p></desc>
</func>
<func><name><ret>int</ret><nametext>enif_have_dirty_schedulers()</nametext></name>
<fsummary>Runtime check for the presence of dirty scheduler threads</fsummary>
<desc>
<p>Check at runtime for the presence of dirty scheduler threads. If the emulator is
built with threading support, dirty scheduler threads are available and
<c>enif_have_dirty_schedulers()</c> returns true. If the emulator was built without
threading support, <c>enif_have_dirty_schedulers()</c> returns false.</p>
<p>If dirty scheduler threads are not available in the emulator, calls to
<c>enif_schedule_dirty_nif</c> and <c>enif_schedule_dirty_nif_finalizer</c> result in
the NIF and finalizer functions being called directly within the calling thread.</p>
<note><p>This function is available only when the emulator is configured with dirty
schedulers enabled. This feature is currently disabled by default. To determine whether
the dirty NIF API is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
</desc>
</func>
<func><name><ret>int</ret><nametext>enif_inspect_binary(ErlNifEnv* env, ERL_NIF_TERM bin_term, ErlNifBinary* bin)</nametext></name>
<fsummary>Inspect the content of a binary</fsummary>
<desc><p>Initialize the structure pointed to by <c>bin</c> with
@@ -777,6 +841,20 @@ typedef enum {
Erlang operators <c>=:=</c> and
<c>=/=</c>.</p></desc>
</func>
<func><name><ret>int</ret><nametext>enif_is_on_dirty_scheduler(ErlNifEnv* env)</nametext></name>
<fsummary>Check to see if executing on a dirty scheduler thread</fsummary>
<desc>
<p>Check to see if the current NIF is executing on a dirty scheduler thread. If the
emulator is built with threading support, calling <c>enif_is_on_dirty_scheduler</c>
from within a dirty NIF returns true. It returns false when the calling NIF is a regular
NIF or a NIF finalizer, both of which run on normal scheduler threads, or when the emulator
is built without threading support.</p>
<note><p>This function is available only when the emulator is configured with dirty
schedulers enabled. This feature is currently disabled by default. To determine whether
the dirty NIF API is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
</desc>
</func>
<func><name><ret>int</ret><nametext>enif_is_pid(ErlNifEnv* env, ERL_NIF_TERM term)</nametext></name>
<fsummary>Determine if a term is a pid</fsummary>
<desc><p>Return true if <c>term</c> is a pid.</p></desc>
@@ -1141,6 +1219,48 @@ typedef enum {
<desc><p>Same as <seealso marker="erl_driver#erl_drv_rwlock_tryrwlock">erl_drv_rwlock_tryrwlock</seealso>.
</p></desc>
</func>
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif(ErlNifEnv* env, int flags, ERL_NIF_TERM (*fp)(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]), int argc, const ERL_NIF_TERM argv[])</nametext></name>
<fsummary>Schedule a dirty NIF for execution</fsummary>
<desc>
<p>Schedule dirty NIF <c>fp</c> to execute a long-running operation. The <c>flags</c>
argument must be set to either <c>ERL_NIF_DIRTY_JOB_CPU_BOUND</c> if the job is expected to
be primarily CPU-bound, or <c>ERL_NIF_DIRTY_JOB_IO_BOUND</c> for jobs that will be
I/O-bound. The <c>argc</c> and <c>argv</c> arguments can either be the originals passed
into the calling NIF, or they can be values created by the calling NIF. The calling
NIF must use the return value of <c>enif_schedule_dirty_nif</c> as its own return value.</p>
<p>Be aware that <c>enif_schedule_dirty_nif</c>, as its name implies, only schedules the
dirty NIF for future execution. The calling NIF does not block waiting for the dirty NIF to
execute and return, which means that the calling NIF can't expect to receive the dirty NIF
return value and use it for further operations.</p>
<p>A dirty NIF may not invoke the <seealso marker="#enif_make_badarg">enif_make_badarg</seealso>
to raise an exception. If it wishes to return an exception, the dirty NIF should pass a
regular result indicating the exception details to its finalizer, and allow the finalizer
to raise the exception on its behalf.</p>
<note><p>This function is available only when the emulator is configured with dirty schedulers
enabled. This feature is currently disabled by default. To determine whether the dirty NIF API
is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
</desc>
</func>
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result, ERL_NIF_TERM (*fp)(ErlNifEnv* env, ERL_NIF_TERM result))</nametext></name>
<fsummary>Schedule a dirty NIF finalizer</fsummary>
<desc>
<p>When a dirty NIF finishes executing, it must schedule a finalizer function to return
its result to the original NIF caller. The dirty NIF passes <c>result</c> as the value it
wants the finalizer to use as the return value. The <c>fp</c> argument is a pointer to the
finalizer function. The NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
enif_dirty_nif_finalizer</seealso> function that can be used as a finalizer that simply
returns its <c>result</c> argument. You are also free to write your own custom finalizer
that uses <c>result</c> to derive a different return value, or ignores <c>result</c>
entirely and returns a completely different value.</p>
<p>Without exception, all dirty NIFs must invoke <c>enif_schedule_dirty_nif_finalizer</c>
to complete their execution.</p>
<note><p>This function is available only when the emulator is configured with dirty
schedulers enabled. This feature is currently disabled by default. To determine whether
the dirty NIF API is available, native code can check to see if the C preprocessor macro
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
</desc>
</func>
<func><name><ret>ErlNifPid *</ret><nametext>enif_self(ErlNifEnv* caller_env, ErlNifPid* pid)</nametext></name>
<fsummary>Get the pid of the calling process.</fsummary>
<desc><p>Initialize the pid variable <c>*pid</c> to represent the
Oops, something went wrong.

0 comments on commit c1c03ae

Please sign in to comment.