Skip to content
This repository

TaskScheduler.hwm default to 1 instead of 0 #1294

Merged
merged 2 commits into from about 2 years ago

2 participants

Min RK Fernando Perez
Min RK
Owner

1 has more predictable/intuitive behavior, if often slower, and thus a more logical default.

closes #1293

added some commits
Min RK change TaskScheduler.hwm default to 1 from 0
0 should be considered an optimization, and 1 is less likely to confuse new users.
c708a85
Min RK document new default hwm value d2d5e36
Fernando Perez fperez merged commit 0cde7de into from
Fernando Perez fperez closed this
Fernando Perez
Owner

Great, thanks. Just merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Jan 19, 2012
Min RK change TaskScheduler.hwm default to 1 from 0
0 should be considered an optimization, and 1 is less likely to confuse new users.
c708a85
Min RK document new default hwm value d2d5e36
This page is out of date. Refresh to see the latest.
16  IPython/parallel/controller/scheduler.py
@@ -131,13 +131,23 @@ class TaskScheduler(SessionFactory):
131 131
 
132 132
     """
133 133
 
134  
-    hwm = Integer(0, config=True, shortname='hwm',
  134
+    hwm = Integer(1, config=True,
135 135
         help="""specify the High Water Mark (HWM) for the downstream
136 136
         socket in the Task scheduler. This is the maximum number
137  
-        of allowed outstanding tasks on each engine."""
  137
+        of allowed outstanding tasks on each engine.
  138
+        
  139
+        The default (1) means that only one task can be outstanding on each
  140
+        engine.  Setting TaskScheduler.hwm=0 means there is no limit, and the
  141
+        engines continue to be assigned tasks while they are working,
  142
+        effectively hiding network latency behind computation, but can result
  143
+        in an imbalance of work when submitting many heterogenous tasks all at
  144
+        once.  Any positive value greater than one is a compromise between the
  145
+        two.
  146
+
  147
+        """
138 148
     )
139 149
     scheme_name = Enum(('leastload', 'pure', 'lru', 'plainrandom', 'weighted', 'twobin'),
140  
-        'leastload', config=True, shortname='scheme', allow_none=False,
  150
+        'leastload', config=True, allow_none=False,
141 151
         help="""select the task scheduler scheme  [default: Python LRU]
142 152
         Options are: 'pure', 'lru', 'plainrandom', 'weighted', 'twobin','leastload'"""
143 153
     )
11  docs/source/parallel/parallel_task.txt
@@ -415,11 +415,11 @@ assigned to an engine at a given time. This limit is set with the
415 415
 .. sourcecode:: python
416 416
 
417 417
     # the most common choices are:
418  
-    c.TaskSheduler.hwm = 0 # (minimal latency, default)
  418
+    c.TaskSheduler.hwm = 0 # (minimal latency, default in IPython ≤ 0.12)
419 419
     # or
420  
-    c.TaskScheduler.hwm = 1 # (most-informed balancing)
  420
+    c.TaskScheduler.hwm = 1 # (most-informed balancing, default in > 0.12)
421 421
 
422  
-The default is 0, or no-limit. That is, there is no limit to the number of
  422
+In IPython ≤ 0.12,the default is 0, or no-limit. That is, there is no limit to the number of
423 423
 tasks that can be outstanding on a given engine. This greatly benefits the
424 424
 latency of execution, because network traffic can be hidden behind computation.
425 425
 However, this means that workload is assigned without knowledge of how long
@@ -429,6 +429,11 @@ effect by setting hwm to a positive integer, 1 being maximum load-balancing (a
429 429
 task will never be waiting if there is an idle engine), and any larger number
430 430
 being a compromise between load-balance and latency-hiding.
431 431
 
  432
+In practice, some users have been confused by having this optimization on by
  433
+default, and the default value has been changed to 1. This can be slower,
  434
+but has more obvious behavior and won't result in assigning too many tasks to
  435
+some engines in heterogeneous cases.
  436
+
432 437
 
433 438
 Pure ZMQ Scheduler
434 439
 ------------------
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.