Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve message about OMP_THREAD_NUM on startup #2946

Merged
merged 6 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions doc/htmldoc/hpc/parallel_computing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -188,12 +188,22 @@ command for this is
Usually, a good choice for `T` is the number of processor cores available
on your machine.

.. note::

In some situations, `oversubscribing` (i.e., to specify a ``local_num_threads`` that is higher than available cores on your machine)
can yield 20-30% improvement in simulation speed. Finding the optimal thread number for a
specific situation might require a bit of experimenting.

In some situations, `oversubscribing` (i.e., to specify a
``local_num_threads`` that is higher than available cores on your
machine) can yield 20–30% improvement in simulation speed. Finding the
optimal thread number for a specific situation might require a bit of
experimenting.

.. admonition:: NEST ignores OMP_NUM_THREADS

NEST ignores ``OMP_NUM_THREADS`` environment
variable, which may be set by mpi4py, Slurm or similar runtime
environments. NEST will always start running on a single thread
until the number of threads is changed by setting either the
``local_num_threads`` or the ``total_num_virtual_procs``
:ref:`kernel attribute<sec_kernel_attributes>`.


Multiprocessing
---------------

Expand Down
2 changes: 1 addition & 1 deletion doc/htmldoc/ref_material/pynest_api/nest.NestModule.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.. _sec:kernel_attributes:
.. _sec_kernel_attributes:

Kernel attributes (nest.NestModule)
===================================
Expand Down
70 changes: 39 additions & 31 deletions nestkernel/vp_manager.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,15 @@ nest::VPManager::initialize()
// of threads used for parallel regions.
omp_set_dynamic( false );
#endif

if ( get_OMP_NUM_THREADS() > 1 )
{
std::string msg = "OMP_NUM_THREADS is set in your environment, but NEST ignores it.\n";
msg += "For details, see the Guide to parallel computing in the NEST Documentation.";

LOG( M_INFO, "VPManager::initialize()", msg );
}

set_num_threads( 1 );
}

Expand All @@ -68,6 +77,20 @@ nest::VPManager::finalize()
{
}

size_t
nest::VPManager::get_OMP_NUM_THREADS() const
{
const char* const omp_num_threads = std::getenv( "OMP_NUM_THREADS" );
if ( omp_num_threads )
{
return std::atoi( omp_num_threads );
}
else
{
return 0;
}
}

void
nest::VPManager::set_status( const DictionaryDatum& d )
{
Expand Down Expand Up @@ -95,22 +118,12 @@ nest::VPManager::set_status( const DictionaryDatum& d )
}
}

if ( force_singlethreading_ and n_threads > 1 )
{
throw BadProperty( "This installation of NEST was built without support for multiple threads." );
}

// We only want to act if new values differ from the old
n_threads_updated = n_threads != get_num_threads();
n_vps_updated = n_vps != get_num_virtual_processes();

if ( n_threads_updated or n_vps_updated )
{
if ( kernel().sp_manager.is_structural_plasticity_enabled() and n_threads > 1 )
{
throw KernelException( "Structural plasticity enabled: multithreading cannot be enabled." );
}

std::vector< std::string > errors;
if ( kernel().node_manager.size() > 0 )
{
Expand All @@ -128,12 +141,16 @@ nest::VPManager::set_status( const DictionaryDatum& d )
{
errors.push_back( "Model defaults were modified" );
}

if ( errors.size() == 1 )
if ( kernel().sp_manager.is_structural_plasticity_enabled() and n_threads > 1 )
{
throw KernelException( errors[ 0 ] + ": number of threads cannot be changed." );
errors.push_back( "Structural plasticity enabled: multithreading cannot be enabled" );
}
if ( errors.size() > 1 )
if ( force_singlethreading_ and n_threads > 1 )
{
errors.push_back( "This installation of NEST does not support for multiple threads" );
}

if ( not errors.empty() )
{
std::string msg = "Number of threads unchanged. Error conditions:";
for ( auto& error : errors )
Expand All @@ -143,6 +160,13 @@ nest::VPManager::set_status( const DictionaryDatum& d )
throw KernelException( msg );
}

if ( get_OMP_NUM_THREADS() > 0 and get_OMP_NUM_THREADS() != n_threads )
{
std::string msg = "OMP_NUM_THREADS is set in your environment, but NEST ignores it.\n";
msg += "For details, see the Guide to parallel computing in the NEST Documentation.";
LOG( M_WARNING, "VPManager::set_status()", msg );
}

kernel().change_number_of_threads( n_threads );
}
}
Expand All @@ -157,23 +181,7 @@ nest::VPManager::get_status( DictionaryDatum& d )
void
nest::VPManager::set_num_threads( size_t n_threads )
{
if ( kernel().sp_manager.is_structural_plasticity_enabled() and n_threads > 1 )
{
throw KernelException( "Multiple threads can not be used if structural plasticity is enabled" );
}

char* omp_num_threads = std::getenv( "OMP_NUM_THREADS" );
if ( omp_num_threads and static_cast< size_t >( std::atoi( omp_num_threads ) ) != n_threads )
{
const std::string tstr = ( n_threads > 1 ) ? "threads" : "thread";
const int ONT = std::atoi( omp_num_threads );
std::string msg = "The new number of threads disagrees with the environment variable OMP_NUM_THREADS.\n";
msg += "NEST only respects the kernel attributes /local_num_threads or /total_num_virtual_procs\n";
msg += String::compose( "and will now use %1 %2 and ignore OMP_NUM_THREADS (set to %3).", n_threads, tstr, ONT );

LOG( M_WARNING, "MPIManager::init_mpi()", msg );
}

assert( not( kernel().sp_manager.is_structural_plasticity_enabled() and n_threads > 1 ) );
n_threads_ = n_threads;

#ifdef _OPENMP
Expand Down
7 changes: 7 additions & 0 deletions nestkernel/vp_manager.h
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,13 @@ class VPManager : public ManagerInterface
*/
size_t get_num_threads() const;

/**
* Get OMP_NUM_THREADS environment variable.
*
* @note Returns 0 if OMP_NUM_THREADS is not set.
*/
size_t get_OMP_NUM_THREADS() const;

/**
* Returns true if the given global node exists on this vp.
*/
Expand Down
Loading