Support for RANDOM language construct in NMODL (#2627)

* Added support for RANDOM construct parsing
* Updated netstim.mod to use new RANDOM construct
* Add tests for usage of RANDOM in HOC and Python
* Documentation for NEURON { RANDOM123 ranvar} and NMODLRandom
* Update NMODL to support CoreNEURON side
* Update NetStim with RANDOM but allow legacy API.
* Update CoreNEURON <-> NEURON for NMODL RANDOM

Co-authored-by: Pramod S Kumbhar <pramod.s.kumbhar@gmail.com>
Co-authored-by: Omar Awile <omar.awile@epfl.ch>

cmake/NeuronFileLists.cmake

@@ -228,6 +228,7 @@ set(NRNIV_FILE_LIST
     multisplit.cpp
     ndatclas.cpp
     netpar.cpp
+    nmodlrandom.cpp
     nonlinz.cpp
     nrncore_write.cpp
     nrncore_write/callbacks/nrncore_callbacks.cpp

docs/cmake_doc/options.rst

@@ -579,6 +579,15 @@ NRN_COVERAGE_FILES:STRING=
 
   ``-DNRN_COVERAGE_FILES="src/nrniv/partrans.cpp;src/nmodl/parsact.cpp;src/nrnpython/nrnpy_hoc.cpp"``
 
+  For a list of all the cpp files changed in a pull request, consider
+  copy/pasting the ``;`` separated list obtained with
+
+  .. code-block:: shell
+
+     a=`git diff --name-only master | grep '\.cpp'`
+     echo $a | sed 's/ /;/g'
+
+
 NRN_SANITIZERS:STRING=
 ----------------------
   Enable some combination of AddressSanitizer, LeakSanitizer, ThreadSanitizer
@@ -589,6 +598,11 @@ NRN_SANITIZERS:STRING=
   ``-DNRN_SANITIZERS=address`` with the use of Python virtual environments; if
   you attempt this then the CMake code should recommend a solution.
 
+  Note: the ``address`` sanitizer also prints leak infornation when a
+  launch exits. That can be avoided with
+
+  ``export ASAN_OPTIONS=detect_leaks=0``
+
 Miscellaneous Rarely used options specific to NEURON:
 =====================================================
 

docs/python/modelspec/programmatic/mechanisms/nmodl2.rst

@@ -44,6 +44,7 @@ Description:
            POINT_PROCESS ... 
            POINTER pointer1, ...
            BBCOREPOINTER bbcore1, ...
+           RANDOM ranvar1, ...
            EXTERNAL external1, ...
            THREADSAFE
            REPRESENTS ontology_id
@@ -410,6 +411,55 @@ Description:
 
     ``TODO``: Add description (?) and existing example mod file (provided by link)
 
+.. _nmodlrandom:
+
+RANDOM
+######
+
+Description:
+    .. code-block::
+
+        NEURON {
+            RANDOM ranvar1, ...
+        }
+
+    These names refer to random variable streams that are automatically
+    associated with nrnran123 generators. Such nrnran123 generators are also used, for example to implement
+    :meth:`Random.Random123`
+    These names are analogous to range variables in that the streams are distinct for every mechanism instance
+    of a POINT_PROCESS, ARTIFICIAL_CELL, or instance of a density mechanism in a segment of a cable section.
+    Each stream exists for the lifetime of the mechanism instance. While a stream exists, its properties can
+    be changed from the interpreter.
+
+    Prior to the introduction of this keyword, random streams required a POINTER variable and
+    fairly elaborate VERBATIM blocks
+    to setup the streams and manage  the stream properties from HOC or Python so that each stream was
+    statistically independent of all other streams.
+
+    From the interpreter, the ranvar1 stream properties are assigned and evaluated using standard
+    range variable syntax where mention of ranvar1 returns a :class:`~NMODLRandom` object that wraps the stream
+    and provides method calls to get and set the three stream ids and the starting sequence number.
+
+    When a stream is instantiated, its identifier triplet is default initialized to
+    (1, :meth:`mpiworldrank <ParallelContext.id_world>`, ++internal_id3)
+    so all streams are statistically independent (at launch time, internal_id3 = 0).
+    However since the identifier triplet depends on the order of
+    construction, it is recommended for parallel simulation reproducibility that triplets be algorithmically specified
+    at the interpreter level. And see :meth:`Random.Random123_globalindex`.
+
+    At present, the list of random_... methods available for use within mod files (outside of VERBATIM blocks) are:
+
+        * random_setseq(ranvar1, uint34_value)
+        * random_setids(ranvar1, id1_uint32, id2_uint32, id3_uint32)
+        * x = random_uniform(ranvar1) : uniform 0 to 1 -- minimum value is 2.3283064e-10 and max value is 1-min
+        * x = random_uniform(ranvar1, min, max)
+        * x = random_negexp(ranvar1) : mean 1.0 -- min value is 2.3283064e-10, max is 22.18071
+        * x = random_negexp(ranvar1, mean)
+        * x = random_normal(ranvar1) : mean 1.0, std 1.0
+        * x = random_normal(ranvar1, mean, std)
+        * x = random_ipick(ranvar1) : range 0 to 2^32-1
+        * x = random_dpick(ranvar1)
+
 
 EXTERNAL
 ########

docs/python/programming/math/random.rst

@@ -768,3 +768,92 @@ Random Class
 
 
 
+----
+
+NMODLRandom Class
+=================
+
+.. class:: NMODLRandom
+
+    Syntax:
+        ``r = point_process.ranvar``
+
+        ``r = section(x).mech.ranvar``
+
+        ``r = section(x).ranvar_mech``
+
+
+    Description:
+        Returns an NMODLRandom wrapper for the nrnran123_State associated with the mechanism
+        :ref:`RANDOM ranvar <nmodlrandom>` variable.
+        Note that an attempt to assign a value to ranvar will raise an error.
+        At present, all mentions of ranvar in the context of a specific mechanism instance return a wrapper for
+        the same nrnran123_State (though the NMODLRandom instances are different).
+
+----
+
+.. method:: NMODLRandom.get_ids
+
+    Syntax:
+        ``vector = r.get_ids()``
+
+    Description:
+        Returns a HOC Vector of size 3 containing the 32 bit id1, id2, id3 of the nrnran123_State
+
+----
+
+.. method:: NMODLRandom.set_ids
+
+    Syntax:
+        ``r = r.set_ids(id1, id2, id3)``
+
+    Description:
+        Sets the 32 bit id1, id2, id3 of the nrnran123_State and returns the same NModlRandom instance.
+
+
+----
+
+.. method:: NMODLRandom.get_seq
+
+    Syntax:
+        ``x = r.get_seq()``
+
+    Description:
+        Returns as a float, the 34 bit sequence position of the nrnran123_State
+
+----
+
+.. method:: NMODLRandom.set_seq
+
+    Syntax:
+        ``r = r.set_seq(x)``
+
+    Description:
+        Sets the 34 bit sequence position of the nrnran123_State. Returns the same NMODLRandom instance.
+
+----
+
+.. method:: NMODLRandom.uniform
+
+    Syntax:
+        ``x = r.uniform()``
+
+    Description:
+        Returns as a float, the uniform random value in the open interval 0 to 1 at the current sequence
+        position of the nrnran123_State (the current sequence position is then incremented by 1)
+        This is, for testing purposes, the only distribution exposed to
+        the interpreter. We don't forsee any practical use of
+        NMODLRandom within the interpreter in regard to sampling. The purpose
+        of NMODLRandom is to allow setting of stream properties for a
+        mod file RANDOM variable. Indeed, if one explicitly constructs an NMODLRandom
+        from the interpreter, then
+
+        .. code-block::
+            python
+
+            from neuron import h
+            r = h.NMODLRandom()
+            print(r.uniform())
+
+            NEURON: NMODLRandom wrapped handle is not valid
+

docs/python/simctrl/bbsavestate.rst

@@ -30,18 +30,24 @@ BBSaveState
           h.stdinit()
           bbss = h.BBSaveState()
           if restore:
-            bbss.restore_test()
+            bbss.restore("temp.dat")
             print(f'after restore t={h.t}')
           else:
             pc.psolve(tstop/2)
-            bbss.save_test()
+            bbss.save("temp.dat")
           pc.psolve(tstop)
 
-    Note that files are saved in a subdirectory called "out" and restored
-    from a subdirectory called "in". An empty "out" folder should be created by
-    the user prior to calling save_test(). A script filter
+    In this case, the entire model state for this MPI rank is in the filename for save and restore.
+
+    If multisplit is involved, or it is desired to reassemble the model cells on a different set of MPI ranks,
+    one instead must use :meth:`BBSaveState.save_test`
+    and :meth:`BBSaveState.restore_test`. This allows reassembly of the multisplit subtrees back into
+    their complete cells to allow different multisplitting and different cell distribution on different ranks.
+    In this case
+    files are saved in a subdirectory called "bbss_out" and restored
+    from a subdirectory called "bbss_in". A script filter
     (see :meth:`BBSaveState.save_test`) is needed to copy and sometimes
-    concatenate files from the out to the in subfolders. These files have
+    concatenate files from the bbss_out to the bbss_in subfolders. These files have
     an ascii format.
 
     BBSaveState has a c++ API that allows one to replace the file reader and
@@ -59,7 +65,7 @@ BBSaveState
     Because a restore clears the event queue and because one cannot call
     finitialize from hoc without vitiating the restore, :meth:`Vector.play` will
     not work unless one calls :meth:`BBSaveState.vector_play_init` after a
-    restore (similarly :func:`frecord` must be called for :meth:`Vector.record` to work.
+    restore (similarly :func:`frecord_init` must be called for :meth:`Vector.record` to work.
     Note that it is necessary that Vector.play use a tvec argument with
     a first element greater than or equal to the restore time.
 
@@ -72,9 +78,10 @@ BBSaveState
        with a base gid.
     4. NetCon.event in Hoc can be used only with NetCon's with a None source.
 
-
-    To allow extra state, such as Random sequence, to be saved for
-    POINT_PROCESS or SUFFIX density nmodl mechanisms,
+    RANDOM variables declared in a mod file NEURON block have their sequence value saved
+    automatically.
+
+    To allow extra state to be saved, eg. when POINTER and BBCOREPOINTER are used to manage objects,
     declare  FUNCTION bbsavestate() within the mechanism.
     That function is called when the
     mechanism instance is saved and restored.
@@ -119,16 +126,16 @@ BBSaveState
 
 
     Description:
-        State of the model is saved in files within the subdirectory, `out`.
-    The file `out/tmp` contains the value of t. Other files have the
+        State of the model is saved in files within the subdirectory, `bbss_out`.
+        The file `bbss_out/tmp` contains the value of t. Other files have the
         filename format tmp.<gid>.<rank> . Only in the case of multisplit
         is it possible to have the same gid in more than one filename. Note
         that the out folder needs to be created by the user prior to a call
         to save_test().
 
         To prepare for a restore, the tmp.<gid>.<rank> files should be copied
-        from the `out` subfolder to a subfolder called `in`, with the filename
-        in/tmp.<gid> . Each file should begin with a first line that specifies
+        from the `bbss_out` subfolder to a subfolder called `bbss_in`, with the filename
+        bbss_in/tmp.<gid> . Each file should begin with a first line that specifies
         the number of files in the `out` folder that had the same gid.
 
         The following out2in.sh script shows how to do this (not particularly
@@ -138,16 +145,16 @@ BBSaveState
           bash
 
           #!/usr/bin/env bash
-          rm -f in/*
-          cat out/tmp > in/tmp
-          for f in out/tmp.*.* ; do
+          rm -f bbss_in/*
+          cat bbss_out/tmp > bbss_in/tmp
+          for f in bbss_out/tmp.*.* ; do
             echo $f
             i=`echo "$f" | sed 's/.*tmp\.\([0-9]*\)\..*/\1/'`
             echo $i
-            if test ! -f in/tmp.$i ; then
-              cnt=`ls out/tmp.$i.* | wc -l`
-              echo $cnt > in/tmp.$i
-              cat out/tmp.$i.* >> in/tmp.$i
+            if test ! -f bbss_in/tmp.$i ; then
+              cnt=`ls bbss_out/tmp.$i.* | wc -l`
+              echo $cnt > bbss_in/tmp.$i
+              cat bbss_out/tmp.$i.* >> bbss_in/tmp.$i
             fi
           done
 
@@ -166,16 +173,40 @@ BBSaveState
 
     Description:
         State of the model is restored from files within the
-        subdirectory, "in". The file "in/tmp" supplies the value of t.
-    Other files have the filename format tmp.<gid> and are read when
+        subdirectory, "bbss_in". The file "bbss_in/tmp" supplies the value of t.
+        Other files have the filename format tmp.<gid> and are read when
         that gid is restored. Note that in a multisplit context, the same
-        "in/tmp.<gid>" file will be read by multiple ranks, but only the state
+        "bbss_in/tmp.<gid>" file will be read by multiple ranks, but only the state
         assocated with sections that exist on a rank will be restored.
 
 ----
 
 
+.. method:: BBSaveState.save
+
+  Syntax:
+    ``.save("filename")``
 
+  Description:
+    Saves the state of the entire model (on this rank). This is simpler to use than the ``save_test``, ``restore_test``
+    pattern but does not work if one has multisplit cells or desires a different distribution of cells on a different
+    number of ranks.
+
+
+----
+
+
+.. method:: BBSaveState.restore
+
+  Syntax:
+    ``.restore("filename")``
+
+  Description:
+    Restores the state of the entire model (on this rank). This is simpler to use than the ``save_test``, ``restore_test``
+    pattern but does not work if one has multisplit cells or desires a different distribution of cells on a different
+    number of ranks.
+
+----
 
 .. method:: BBSaveState.ignore