Skip to content

Commit c1c03ae

Browse files
vinoskirickard-green
authored andcommitted
initial support for dirty schedulers and dirty NIFs
Add initial support for dirty schedulers. There are two types of dirty schedulers: CPU schedulers and I/O schedulers. By default, there are as many dirty CPU schedulers as there are normal schedulers and as many dirty CPU schedulers online as normal schedulers online. There are 10 dirty I/O schedulers (similar to the choice of 10 as the default for async threads). By default, dirty schedulers are disabled and conditionally compiled out. To enable them, you must pass --enable-dirty-schedulers to the top-level configure script when building Erlang/OTP. Current dirty scheduler support requires the emulator to be built with SMP support. This restriction will be lifted in the future. You can specify the number of dirty schedulers with the command-line options +SDcpu (for dirty CPU schedulers) and +SDio (for dirty I/O schedulers). The +SDcpu option is similar to the +S option in that it takes two numbers separated by a colon: C1:C2, where C1 specifies the number of dirty schedulers available and C2 specifies the number of dirty schedulers online. The +SDPcpu option allows numbers of dirty CPU schedulers available and dirty CPU schedulers online to be specified as percentages, similar to the existing +SP option for normal schedulers. The number of dirty CPU schedulers created and dirty CPU schedulers online may not exceed the number of normal schedulers created and normal schedulers online, respectively. The +SDio option takes only a single number specifying the number of dirty I/O schedulers available and online. There is no support yet for programmatically changing at run time the number of dirty CPU schedulers online via erlang:system_flag/2. Also, changing the number of normal schedulers online via erlang:system_flag(schedulers_online, NewSchedulersOnline) should ensure that there are no more dirty CPU schedulers than normal schedulers, but this is not yet implemented. You can retrieve the number of dirty schedulers by passing dirty_cpu_schedulers, dirty_cpu_schedulers_online, or dirty_io_schedulers to erlang:system_info/1. Currently only NIFs are able to access dirty scheduler functionality. Neither drivers nor BIFs currently support dirty schedulers. This restriction will be addressed in the future. If dirty scheduler support is present in the runtime, the initial status line Erlang prints before presenting its interactive prompt will include the indicator "[ds:C1:C2:I]" where "ds" indicates "dirty schedulers", "C1" indicates the number of dirty CPU schedulers available, "C2" indicates the number of dirty CPU schedulers online, and "I" indicates the number of dirty I/O schedulers. Document The dirty NIF API in the erl_nif man page. The API closely follows Rickard Green's presentation slides from his talk "Future Extensions to the Native Interface", presented at the 2011 Erlang Factory held in the San Francisco Bay Area. Rickard's slides are available online at http://bit.ly/1m34UHB . Document the new erl command-line options, the additions to erlang:system_info/1, and also add the erlang:system_flag/2 dirty scheduler documentation even though it's not yet implemented. To determine whether the dirty NIF API is available, native code can check to see whether the C preprocessor macro ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined. To check if dirty schedulers are available at run time, native code can call the boolean enif_have_dirty_schedulers() function, and Erlang code can call erlang:system_info(dirty_cpu_schedulers), which raises badarg if no dirty scheduler support is available. Add a simple dirty NIF test to the emulator NIF suite.
1 parent aa1b402 commit c1c03ae

File tree

24 files changed

+1459
-202
lines changed

24 files changed

+1459
-202
lines changed

HOWTO/INSTALL.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,11 @@ Some of the available `configure` options are:
315315
no automatic dependency handling inbetween applications. So if you disable
316316
an application that another depends on, you also have to disable the
317317
dependant application.
318+
* `--enable-dirty-schedulers` - Enable the **experimental** dirty schedulers
319+
functionality. Note that the dirty schedulers functionality is experimental,
320+
and **not supported**. This functionality **will** be subject to backward
321+
incompatible changes. Note that you should **not** enable the dirty scheduler
322+
functionality on production systems. It is only provided for testing.
318323

319324
If you or your system has special requirements please read the `Makefile` for
320325
additional configuration information.

configure.in

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,9 @@ AC_ARG_ENABLE(threads,
196196
AS_HELP_STRING([--enable-threads], [enable async thread support])
197197
AS_HELP_STRING([--disable-threads], [disable async thread support]))
198198

199+
AC_ARG_ENABLE(dirty-schedulers,
200+
AS_HELP_STRING([--enable-dirty-schedulers], [enable dirty scheduler support]))
201+
199202
AC_ARG_ENABLE(halfword-emulator,
200203
AS_HELP_STRING([--enable-halfword-emulator],
201204
[enable halfword emulator (only for 64bit builds)]))

erts/configure.in

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ ERL_XCOMP_SYSROOT_INIT
101101

102102
AC_ISC_POSIX
103103

104-
AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in)
104+
AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_header_config.h:include/internal/ethread_header_config.h.in include/$host/erl_int_sizes_config.h:include/erl_int_sizes_config.h.in include/$host/erl_native_features_config.h:include/erl_native_features_config.h.in)
105105
dnl ----------------------------------------------------------------------
106106
dnl Optional features.
107107
dnl ----------------------------------------------------------------------
@@ -123,6 +123,7 @@ AS_HELP_STRING([--enable-bootstrap-only],
123123
with_ssl_zlib=no
124124
enable_hipe=no
125125
enable_sctp=no
126+
enable_dirty_schedulers=no
126127
fi
127128
])
128129

@@ -134,6 +135,13 @@ AS_HELP_STRING([--disable-threads], [disable async thread support]),
134135
*) enable_threads=yes ;;
135136
esac ], enable_threads=unknown)
136137

138+
AC_ARG_ENABLE(dirty-schedulers,
139+
AS_HELP_STRING([--enable-dirty-schedulers], [enable dirty scheduler support]),
140+
[ case "$enableval" in
141+
no) enable_dirty_schedulers=no ;;
142+
*) enable_dirty_schedulers=yes ;;
143+
esac ], enable_dirty_schedulers=no)
144+
137145
AC_ARG_ENABLE(halfword-emulator,
138146
AS_HELP_STRING([--enable-halfword-emulator],
139147
[enable halfword emulator (only for 64bit builds)]),
@@ -1222,6 +1230,13 @@ esac
12221230
if test $emu_threads != yes; then
12231231
enable_lock_check=no
12241232
enable_lock_count=no
1233+
AC_MSG_CHECKING(whether dirty schedulers should be enabled)
1234+
if test "x$enable_dirty_schedulers" != "xno"; then
1235+
AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support])
1236+
AC_MSG_RESULT(yes)
1237+
else
1238+
AC_MSG_RESULT(no)
1239+
fi
12251240
else
12261241
# Threads enabled for emulator
12271242
EMU_THR_LIB_NAME=$ETHR_LIB_NAME
@@ -1240,7 +1255,17 @@ else
12401255
if test "x$enable_lock_count" != "xno"; then
12411256
EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_ENABLE_LOCK_COUNT"
12421257
fi
1243-
1258+
1259+
AC_MSG_CHECKING(whether dirty schedulers should be enabled)
1260+
if test "x$enable_dirty_schedulers" != "xno"; then
1261+
EMU_THR_DEFS="$EMU_THR_DEFS -DERTS_DIRTY_SCHEDULERS"
1262+
AC_DEFINE(ERTS_DIRTY_SCHEDULERS, 1, [Define if the emulator supports dirty schedulers])
1263+
AC_DEFINE(ERL_NIF_DIRTY_SCHEDULER_SUPPORT, 1, [Dirty scheduler support])
1264+
AC_MSG_RESULT(yes)
1265+
else
1266+
AC_MSG_RESULT(no)
1267+
fi
1268+
12441269
disable_child_waiter_thread=no
12451270
case $host_os in
12461271
solaris*)

erts/doc/src/erl.xml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -792,6 +792,54 @@
792792
SMP support enabled (see the <seealso marker="#smp">-smp</seealso>
793793
flag).</p>
794794
</item>
795+
<tag><marker id="+SDcpu"><c><![CDATA[+SDcpu DirtyCPUSchedulers:DirtyCPUSchedulersOnline]]></c></marker></tag>
796+
<item>
797+
<p>Sets the number of dirty CPU scheduler threads to create and dirty
798+
CPU scheduler threads to set online when threading support has been
799+
enabled. The maximum for both values is 1024, and each value is further
800+
limited by the settings for normal schedulers: the number of dirty CPU
801+
scheduler threads created cannot exceed the number of normal scheduler
802+
threads created, and the number of dirty CPU scheduler threads online
803+
cannot exceed the number of normal scheduler threads online (see the
804+
<seealso marker="#+S">+S</seealso> and <seealso marker="#+SP">+SP</seealso>
805+
flags for more details). By default, the number of dirty CPU scheduler
806+
threads created equals the number of normal scheduler threads created,
807+
and the number of dirty CPU scheduler threads online equals the number
808+
of normal scheduler threads online. <c>DirtyCPUSchedulers</c> may be
809+
omitted if <c>:DirtyCPUSchedulersOnline</c> is not and vice versa. The
810+
number of dirty CPU schedulers online can be changed at run time via
811+
<seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
812+
</p>
813+
<p>This option is ignored if the emulator doesn't have threading support
814+
enabled. Currently, <em>this option is experimental</em> and is supported only
815+
if the emulator was configured and built with support for dirty schedulers
816+
enabled (it's disabled by default).
817+
</p>
818+
</item>
819+
<tag><marker id="+SDPcpu"><c><![CDATA[+SDPcpu DirtyCPUSchedulersPercentage:DirtyCPUSchedulersOnlinePercentage]]></c></marker></tag>
820+
<item>
821+
<p>Similar to <seealso marker="#+SDcpu">+SDcpu</seealso> but uses percentages to set the
822+
number of dirty CPU scheduler threads to create and number of dirty CPU scheduler threads
823+
to set online when threading support has been enabled. Specified values must be greater
824+
than 0. For example, <c>+SDPcpu 50:25</c> sets the number of dirty CPU scheduler threads
825+
to 50% of the logical processors configured and the number of dirty CPU scheduler threads
826+
online to 25% of the logical processors available. <c>DirtyCPUSchedulersPercentage</c> may
827+
be omitted if <c>:DirtyCPUSchedulersOnlinePercentage</c> is not and vice versa. The
828+
number of dirty CPU schedulers online can be changed at run time via
829+
<seealso marker="erlang#system_flag_dirty_cpu_schedulers_online">erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline)</seealso>.
830+
</p>
831+
<p>This option interacts with <seealso marker="#+SDcpu">+SDcpu</seealso> settings.
832+
For example, on a system with 8 logical cores configured and 8 logical cores available,
833+
the combination of the options <c>+SDcpu 4:4 +SDPcpu 50:25</c> (in either order) results
834+
in 2 dirty CPU scheduler threads (50% of 4) and 1 dirty CPU scheduler thread online (25% of 4).
835+
</p>
836+
<p>This option is ignored if the emulator doesn't have threading support
837+
enabled. Currently, <em>this option is experimental</em> and is supported only
838+
if the emulator was configured and built with support for dirty schedulers
839+
enabled (it's disabled by default).
840+
</p>
841+
</item>
842+
<tag><marker id="+SDio"><c><![CDATA[+SDio IOSchedulers]]></c></marker></tag>
795843
<tag><c><![CDATA[+sFlag Value]]></c></tag>
796844
<item>
797845
<p>Scheduling specific flags.</p>

erts/doc/src/erl_nif.xml

Lines changed: 121 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,11 @@ ok
181181
to dispatch the work to another thread, return
182182
from the native function, and wait for the result. The thread can send
183183
the result back to the calling thread using message passing. Information
184-
about thread primitives can be found below.</p>
184+
about thread primitives can be found below. If you have built your system
185+
with <em>the currently experimental</em> support for dirty schedulers,
186+
you may want to try out this functionality by dispatching the work to a
187+
<seealso marker="#dirty_nifs">dirty NIF</seealso>,
188+
which does not have the same duration restriction as a normal NIF.</p>
185189
</description>
186190
<section>
187191
<title>FUNCTIONALITY</title>
@@ -312,6 +316,38 @@ ok
312316
<p>The library initialization callbacks <c>load</c>, <c>reload</c> and
313317
<c>upgrade</c> are all thread-safe even for shared state data.</p>
314318
</item>
319+
<tag>Dirty NIFs</tag>
320+
<item><p><marker id="dirty_nifs"/><em>Note that the dirty NIF functionality
321+
is experimental</em> and that you have to enable support for dirty
322+
schedulers when building OTP in order to try the functionality out. Native functions
323+
<seealso marker="#lengthy_work">
324+
must normally run quickly</seealso>, as explained earlier in this document. They
325+
generally should execute for no more than a millisecond. But not all native functions
326+
can execute so quickly; for example, functions that encrypt large blocks of data or
327+
perform lengthy file system operations can often run for tens of seconds or more.</p>
328+
<p>A NIF that cannot execute in a millisecond or less is called a "dirty NIF" since
329+
it performs work that the Erlang runtime cannot handle cleanly. Applications
330+
that make use of such functions must indicate to the runtime that the functions are
331+
dirty so they can be handled specially. To schedule a dirty NIF for execution, the
332+
application calls <seealso marker="#enif_schedule_dirty_nif">enif_schedule_dirty_nif</seealso>,
333+
passing to it a pointer to the dirty NIF to be executed and indicating with a flag
334+
argument whether it expects the operation to be CPU-bound or I/O-bound.</p>
335+
<p>All dirty NIFs must ultimately invoke the <seealso marker="#enif_schedule_dirty_nif_finalizer">
336+
enif_schedule_dirty_nif_finalizer</seealso> as their final action, passing to it the
337+
result they wish to return to the original caller. A finalizer function can either
338+
receive the result and return it directly, or it can return a different value instead.
339+
For convenience, the NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
340+
enif_dirty_nif_finalizer</seealso> function that applications can use as a finalizer;
341+
it simply returns its result argument.</p>
342+
<note><p>Dirty NIF support is available only when the emulator is configured with dirty
343+
schedulers enabled. This feature is currently disabled by default. To determine whether
344+
the dirty NIF API is available, native code can check to see if the C preprocessor macro
345+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined. Also, if the Erlang runtime was built
346+
without threading support, dirty schedulers are disabled. To check at runtime for the presence
347+
of dirty scheduler threads, code can call the <seealso marker="#enif_have_dirty_schedulers"><c>
348+
enif_have_dirty_schedulers()</c></seealso> API function, which returns true if dirty
349+
scheduler threads are present, false otherwise.</p></note>
350+
</item>
315351
</taglist>
316352
</section>
317353
<section>
@@ -610,6 +646,18 @@ typedef enum {
610646
See also the <seealso marker="#WARNING">warning</seealso> text at the beginning of this document.</p>
611647
</desc>
612648
</func>
649+
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result)</nametext></name>
650+
<fsummary>Simple dirty NIF result finalizer</fsummary>
651+
<desc>
652+
<p>A convenience function that a dirty NIF can use as a finalizer that simply
653+
return its <c>result</c> argument as its return value. This function is provided
654+
for dirty NIFs with results that should be returned directly to the original caller.</p>
655+
<note><p>This function is available only when the emulator is configured with dirty
656+
schedulers enabled. This feature is currently disabled by default. To determine whether
657+
the dirty NIF API is available, native code can check to see if the C preprocessor macro
658+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
659+
</desc>
660+
</func>
613661
<func><name><ret>int</ret><nametext>enif_equal_tids(ErlNifTid tid1, ErlNifTid tid2)</nametext></name>
614662
<fsummary></fsummary>
615663
<desc><p>Same as <seealso marker="erl_driver#erl_drv_equal_tids">erl_drv_equal_tids</seealso>.
@@ -730,6 +778,22 @@ typedef enum {
730778
and return true, or return false if <c>term</c> is not an unsigned integer or is
731779
outside the bounds of type <c>unsigned long</c>.</p></desc>
732780
</func>
781+
<func><name><ret>int</ret><nametext>enif_have_dirty_schedulers()</nametext></name>
782+
<fsummary>Runtime check for the presence of dirty scheduler threads</fsummary>
783+
<desc>
784+
<p>Check at runtime for the presence of dirty scheduler threads. If the emulator is
785+
built with threading support, dirty scheduler threads are available and
786+
<c>enif_have_dirty_schedulers()</c> returns true. If the emulator was built without
787+
threading support, <c>enif_have_dirty_schedulers()</c> returns false.</p>
788+
<p>If dirty scheduler threads are not available in the emulator, calls to
789+
<c>enif_schedule_dirty_nif</c> and <c>enif_schedule_dirty_nif_finalizer</c> result in
790+
the NIF and finalizer functions being called directly within the calling thread.</p>
791+
<note><p>This function is available only when the emulator is configured with dirty
792+
schedulers enabled. This feature is currently disabled by default. To determine whether
793+
the dirty NIF API is available, native code can check to see if the C preprocessor macro
794+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
795+
</desc>
796+
</func>
733797
<func><name><ret>int</ret><nametext>enif_inspect_binary(ErlNifEnv* env, ERL_NIF_TERM bin_term, ErlNifBinary* bin)</nametext></name>
734798
<fsummary>Inspect the content of a binary</fsummary>
735799
<desc><p>Initialize the structure pointed to by <c>bin</c> with
@@ -777,6 +841,20 @@ typedef enum {
777841
Erlang operators <c>=:=</c> and
778842
<c>=/=</c>.</p></desc>
779843
</func>
844+
<func><name><ret>int</ret><nametext>enif_is_on_dirty_scheduler(ErlNifEnv* env)</nametext></name>
845+
<fsummary>Check to see if executing on a dirty scheduler thread</fsummary>
846+
<desc>
847+
<p>Check to see if the current NIF is executing on a dirty scheduler thread. If the
848+
emulator is built with threading support, calling <c>enif_is_on_dirty_scheduler</c>
849+
from within a dirty NIF returns true. It returns false when the calling NIF is a regular
850+
NIF or a NIF finalizer, both of which run on normal scheduler threads, or when the emulator
851+
is built without threading support.</p>
852+
<note><p>This function is available only when the emulator is configured with dirty
853+
schedulers enabled. This feature is currently disabled by default. To determine whether
854+
the dirty NIF API is available, native code can check to see if the C preprocessor macro
855+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
856+
</desc>
857+
</func>
780858
<func><name><ret>int</ret><nametext>enif_is_pid(ErlNifEnv* env, ERL_NIF_TERM term)</nametext></name>
781859
<fsummary>Determine if a term is a pid</fsummary>
782860
<desc><p>Return true if <c>term</c> is a pid.</p></desc>
@@ -1141,6 +1219,48 @@ typedef enum {
11411219
<desc><p>Same as <seealso marker="erl_driver#erl_drv_rwlock_tryrwlock">erl_drv_rwlock_tryrwlock</seealso>.
11421220
</p></desc>
11431221
</func>
1222+
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif(ErlNifEnv* env, int flags, ERL_NIF_TERM (*fp)(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]), int argc, const ERL_NIF_TERM argv[])</nametext></name>
1223+
<fsummary>Schedule a dirty NIF for execution</fsummary>
1224+
<desc>
1225+
<p>Schedule dirty NIF <c>fp</c> to execute a long-running operation. The <c>flags</c>
1226+
argument must be set to either <c>ERL_NIF_DIRTY_JOB_CPU_BOUND</c> if the job is expected to
1227+
be primarily CPU-bound, or <c>ERL_NIF_DIRTY_JOB_IO_BOUND</c> for jobs that will be
1228+
I/O-bound. The <c>argc</c> and <c>argv</c> arguments can either be the originals passed
1229+
into the calling NIF, or they can be values created by the calling NIF. The calling
1230+
NIF must use the return value of <c>enif_schedule_dirty_nif</c> as its own return value.</p>
1231+
<p>Be aware that <c>enif_schedule_dirty_nif</c>, as its name implies, only schedules the
1232+
dirty NIF for future execution. The calling NIF does not block waiting for the dirty NIF to
1233+
execute and return, which means that the calling NIF can't expect to receive the dirty NIF
1234+
return value and use it for further operations.</p>
1235+
<p>A dirty NIF may not invoke the <seealso marker="#enif_make_badarg">enif_make_badarg</seealso>
1236+
to raise an exception. If it wishes to return an exception, the dirty NIF should pass a
1237+
regular result indicating the exception details to its finalizer, and allow the finalizer
1238+
to raise the exception on its behalf.</p>
1239+
<note><p>This function is available only when the emulator is configured with dirty schedulers
1240+
enabled. This feature is currently disabled by default. To determine whether the dirty NIF API
1241+
is available, native code can check to see if the C preprocessor macro
1242+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
1243+
</desc>
1244+
</func>
1245+
<func><name><ret>ERL_NIF_TERM</ret><nametext>enif_schedule_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result, ERL_NIF_TERM (*fp)(ErlNifEnv* env, ERL_NIF_TERM result))</nametext></name>
1246+
<fsummary>Schedule a dirty NIF finalizer</fsummary>
1247+
<desc>
1248+
<p>When a dirty NIF finishes executing, it must schedule a finalizer function to return
1249+
its result to the original NIF caller. The dirty NIF passes <c>result</c> as the value it
1250+
wants the finalizer to use as the return value. The <c>fp</c> argument is a pointer to the
1251+
finalizer function. The NIF API provides the <seealso marker="#enif_dirty_nif_finalizer">
1252+
enif_dirty_nif_finalizer</seealso> function that can be used as a finalizer that simply
1253+
returns its <c>result</c> argument. You are also free to write your own custom finalizer
1254+
that uses <c>result</c> to derive a different return value, or ignores <c>result</c>
1255+
entirely and returns a completely different value.</p>
1256+
<p>Without exception, all dirty NIFs must invoke <c>enif_schedule_dirty_nif_finalizer</c>
1257+
to complete their execution.</p>
1258+
<note><p>This function is available only when the emulator is configured with dirty
1259+
schedulers enabled. This feature is currently disabled by default. To determine whether
1260+
the dirty NIF API is available, native code can check to see if the C preprocessor macro
1261+
<c>ERL_NIF_DIRTY_SCHEDULER_SUPPORT</c> is defined.</p></note>
1262+
</desc>
1263+
</func>
11441264
<func><name><ret>ErlNifPid *</ret><nametext>enif_self(ErlNifEnv* caller_env, ErlNifPid* pid)</nametext></name>
11451265
<fsummary>Get the pid of the calling process.</fsummary>
11461266
<desc><p>Initialize the pid variable <c>*pid</c> to represent the

0 commit comments

Comments
 (0)