Skip to content
This repository

Backgroundjobs #856

Merged
merged 4 commits into from over 2 years ago

2 participants

Fernando Perez Min RK
Fernando Perez
Owner

Updates the backgroundjobs module and adds both a test suite and an illustrative notebook. I actually realized, in the conversation on gh-844, that this was pretty useful code.

I don't have time now to complete bringing the %bg magic back, but that would be pretty easy. But otherwise I think this is useful code.

BTW, right now if one moves to a new cell with a background job printing in an old one, and execute the new cell, futher output from the old bg job shows up in the new cell. It seems to me that we should be able to track the origin of the messages and ensure that they continue appearing in their cell of origin. But I've run out of time to dig into that part of the code, so unless @minrk or @ellisonbg knows off the top of your head how to fix that, we can leave it for later. But it would make for very neat and useful patterns, leaving tasks in the background that continue updating a given cell...

added some commits October 10, 2011
Fernando Perez Update backgroundjobs lib to current APIs.
This still doesn't bring the %bg magic back in, but it ensures the
basic library works with current code.
70cdc43
Fernando Perez Cleanup the API of backgroundjobs to be more convenient to use.
Add support for automatic printing of all tracebacks.

The script at the bottom was removed, as a separate notebook will
illustrate use in a more convenient fashion.
d15033a
Fernando Perez Add a notebook illustrating the backgroundjobs library. 7fbfb3c
Fernando Perez Add basic test suite to background jobs library. eac11eb
Min RK
Owner

This looks good to me.

I don't think attaching output to cells other than the current one is very simple. The issue is tracking the parent message, which we do at the beginning of execute requests here. Any print/display calls made after that will be attached to that parent, including those made in threads which launched earlier. What you want would require that background threads are not affected by later calls to set_parent(). This essentially means that in the scope of the Thread, shell.displayhook, shell.display_pub, sys.stdout, and sys.stderr must all somehow be different objects from those in the regular scope. I don't know how that's really possible, but the only alternative I see would be to add inspect calls to IOStream.write, which would walk up into some identifying frame to get the appropriate parent on every single call, and have to add extra flushes in case it has pending messages from a previous parent, to prevent collisions.

With the current model, Threads have no attachment whatsoever to the cell that launched them. By launching something in a thread, you are divorcing that code from the cell in which it launched.

Fernando Perez
Owner

Thanks for the review, @minrk. Good analysis on the thread/stdout question, thanks. We'll just table that idea for now: it's not like it's any worse than threads at a terminal, so I have no problem moving on as-is. I'll merge this now then, one less thing to linger.

Fernando Perez fperez merged commit c40702c into from October 16, 2011
Fernando Perez fperez closed this October 16, 2011
Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 4 unique commits by 1 author.

Oct 10, 2011
Fernando Perez Update backgroundjobs lib to current APIs.
This still doesn't bring the %bg magic back in, but it ensures the
basic library works with current code.
70cdc43
Fernando Perez Cleanup the API of backgroundjobs to be more convenient to use.
Add support for automatic printing of all tracebacks.

The script at the bottom was removed, as a separate notebook will
illustrate use in a more convenient fashion.
d15033a
Fernando Perez Add a notebook illustrating the backgroundjobs library. 7fbfb3c
Fernando Perez Add basic test suite to background jobs library. eac11eb
This page is out of date. Refresh to see the latest.
234  IPython/lib/backgroundjobs.py
@@ -17,6 +17,9 @@
17 17
 
18 18
 (although ultimately no code from this text was used, as IPython's system is a
19 19
 separate implementation).
  20
+
  21
+An example notebook is provided in our documentation illustrating interactive
  22
+use of the system.
20 23
 """
21 24
 
22 25
 #*****************************************************************************
@@ -33,7 +36,8 @@
33 36
 from IPython.core.ultratb import AutoFormattedTB
34 37
 from IPython.utils.warn import warn, error
35 38
 
36  
-class BackgroundJobManager:
  39
+
  40
+class BackgroundJobManager(object):
37 41
     """Class to manage a pool of backgrounded threaded jobs.
38 42
 
39 43
     Below, we assume that 'jobs' is a BackgroundJobManager instance.
@@ -52,7 +56,7 @@ class BackgroundJobManager:
52 56
 
53 57
       jobs.remove(N) -> remove (finished) job N
54 58
 
55  
-      jobs.flush_finished() -> remove all finished jobs
  59
+      jobs.flush() -> remove all finished jobs
56 60
       
57 61
     As a convenience feature, BackgroundJobManager instances provide the
58 62
     utility result and traceback methods which retrieve the corresponding
@@ -63,17 +67,16 @@ class BackgroundJobManager:
63 67
 
64 68
     While this appears minor, it allows you to use tab completion
65 69
     interactively on the job manager instance.
66  
-
67  
-    In interactive mode, IPython provides the magic fuction %bg for quick
68  
-    creation of backgrounded expression-based jobs. Type bg? for details."""
  70
+    """
69 71
 
70 72
     def __init__(self):
71  
-        # Lists for job management
72  
-        self.jobs_run  = []
73  
-        self.jobs_comp = []
74  
-        self.jobs_dead = []
  73
+        # Lists for job management, accessed via a property to ensure they're
  74
+        # up to date.x
  75
+        self._running  = []
  76
+        self._completed = []
  77
+        self._dead = []
75 78
         # A dict of all jobs, so users can easily access any of them
76  
-        self.jobs_all = {}
  79
+        self.all = {}
77 80
         # For reporting
78 81
         self._comp_report = []
79 82
         self._dead_report = []
@@ -83,7 +86,22 @@ def __init__(self):
83 86
         self._s_completed = BackgroundJobBase.stat_completed_c
84 87
         self._s_dead      = BackgroundJobBase.stat_dead_c
85 88
 
86  
-    def new(self,func_or_exp,*args,**kwargs):
  89
+    @property
  90
+    def running(self):
  91
+        self._update_status()
  92
+        return self._running
  93
+
  94
+    @property
  95
+    def dead(self):
  96
+        self._update_status()
  97
+        return self._dead
  98
+
  99
+    @property
  100
+    def completed(self):
  101
+        self._update_status()
  102
+        return self._completed
  103
+
  104
+    def new(self, func_or_exp, *args, **kwargs):
87 105
         """Add a new background job and start it in a separate thread.
88 106
 
89 107
         There are two types of jobs which can be created:
@@ -106,14 +124,14 @@ def new(self,func_or_exp,*args,**kwargs):
106 124
         2. Jobs given a function object, optionally passing additional
107 125
         positional arguments:
108 126
 
109  
-          job_manager.new(myfunc,x,y)
  127
+          job_manager.new(myfunc, x, y)
110 128
 
111 129
         The function is called with the given arguments.
112 130
 
113 131
         If you need to pass keyword arguments to your function, you must
114 132
         supply them as a dict named kw:
115 133
 
116  
-          job_manager.new(myfunc,x,y,kw=dict(z=1))
  134
+          job_manager.new(myfunc, x, y, kw=dict(z=1))
117 135
 
118 136
         The reason for this assymmetry is that the new() method needs to
119 137
         maintain access to its own keywords, and this prevents name collisions
@@ -149,7 +167,7 @@ def new(self,func_or_exp,*args,**kwargs):
149 167
         if callable(func_or_exp):
150 168
             kw  = kwargs.get('kw',{})
151 169
             job = BackgroundJobFunc(func_or_exp,*args,**kw)
152  
-        elif isinstance(func_or_exp,basestring):
  170
+        elif isinstance(func_or_exp, basestring):
153 171
             if not args:
154 172
                 frame = sys._getframe(1)
155 173
                 glob, loc = frame.f_globals, frame.f_locals
@@ -158,30 +176,28 @@ def new(self,func_or_exp,*args,**kwargs):
158 176
             elif len(args)==2:
159 177
                 glob,loc = args
160 178
             else:
161  
-                raise ValueError,\
162  
-                      'Expression jobs take at most 2 args (globals,locals)'
163  
-            job = BackgroundJobExpr(func_or_exp,glob,loc)
  179
+                raise ValueError(
  180
+                      'Expression jobs take at most 2 args (globals,locals)')
  181
+            job = BackgroundJobExpr(func_or_exp, glob, loc)
164 182
         else:
165  
-            raise
166  
-        jkeys = self.jobs_all.keys()
167  
-        if jkeys:
168  
-            job.num = max(jkeys)+1
169  
-        else:
170  
-            job.num = 0
171  
-        self.jobs_run.append(job)
172  
-        self.jobs_all[job.num] = job
  183
+            raise TypeError('invalid args for new job')
  184
+        
  185
+        job.num = len(self.all)+1 if self.all else 0
  186
+        self.running.append(job)
  187
+        self.all[job.num] = job
173 188
         print 'Starting job # %s in a separate thread.' % job.num
174 189
         job.start()
175 190
         return job
176 191
 
177  
-    def __getitem__(self,key):
178  
-        return self.jobs_all[key]
  192
+    def __getitem__(self, job_key):
  193
+        num = job_key if isinstance(job_key, int) else job_key.num
  194
+        return self.all[num]
179 195
 
180 196
     def __call__(self):
181 197
         """An alias to self.status(),
182 198
 
183 199
         This allows you to simply call a job manager instance much like the
184  
-        Unix jobs shell command."""
  200
+        Unix `jobs` shell command."""
185 201
 
186 202
         return self.status()
187 203
 
@@ -189,29 +205,34 @@ def _update_status(self):
189 205
         """Update the status of the job lists.
190 206
 
191 207
         This method moves finished jobs to one of two lists:
192  
-          - self.jobs_comp: jobs which completed successfully
193  
-          - self.jobs_dead: jobs which finished but died.
  208
+          - self.completed: jobs which completed successfully
  209
+          - self.dead: jobs which finished but died.
194 210
 
195 211
         It also copies those jobs to corresponding _report lists.  These lists
196 212
         are used to report jobs completed/dead since the last update, and are
197 213
         then cleared by the reporting function after each call."""
198  
-        
199  
-        run,comp,dead = self._s_running,self._s_completed,self._s_dead
200  
-        jobs_run = self.jobs_run
201  
-        for num in range(len(jobs_run)):
202  
-            job  = jobs_run[num]
  214
+
  215
+        # Status codes
  216
+        srun, scomp, sdead = self._s_running, self._s_completed, self._s_dead
  217
+        # State lists, use the actual lists b/c the public names are properties
  218
+        # that call this very function on access
  219
+        running, completed, dead = self._running, self._completed, self._dead
  220
+
  221
+        # Now, update all state lists
  222
+        for num, job in enumerate(running):
203 223
             stat = job.stat_code
204  
-            if stat == run:
  224
+            if stat == srun:
205 225
                 continue
206  
-            elif stat == comp:
207  
-                self.jobs_comp.append(job)
  226
+            elif stat == scomp:
  227
+                completed.append(job)
208 228
                 self._comp_report.append(job)
209  
-                jobs_run[num] = False
210  
-            elif stat == dead:
211  
-                self.jobs_dead.append(job)
  229
+                running[num] = False
  230
+            elif stat == sdead:
  231
+                dead.append(job)
212 232
                 self._dead_report.append(job)
213  
-                jobs_run[num] = False
214  
-        self.jobs_run = filter(None,self.jobs_run)
  233
+                running[num] = False
  234
+        # Remove dead/completed jobs from running list
  235
+        running[:] = filter(None, running)
215 236
 
216 237
     def _group_report(self,group,name):
217 238
         """Report summary for a given job group.
@@ -246,7 +267,7 @@ def _status_new(self):
246 267
         which have finished since the last time it was called."""
247 268
 
248 269
         self._update_status()
249  
-        new_comp = self._group_report(self._comp_report,'Completed')
  270
+        new_comp = self._group_report(self._comp_report, 'Completed')
250 271
         new_dead = self._group_report(self._dead_report,
251 272
                                       'Dead, call jobs.traceback() for details')
252 273
         self._comp_report[:] = []
@@ -257,9 +278,9 @@ def status(self,verbose=0):
257 278
         """Print a status of all jobs currently being managed."""
258 279
 
259 280
         self._update_status()
260  
-        self._group_report(self.jobs_run,'Running')
261  
-        self._group_report(self.jobs_comp,'Completed')
262  
-        self._group_report(self.jobs_dead,'Dead')
  281
+        self._group_report(self.running,'Running')
  282
+        self._group_report(self.completed,'Completed')
  283
+        self._group_report(self.dead,'Dead')
263 284
         # Also flush the report queues
264 285
         self._comp_report[:] = []
265 286
         self._dead_report[:] = []
@@ -268,7 +289,7 @@ def remove(self,num):
268 289
         """Remove a finished (completed or dead) job."""
269 290
 
270 291
         try:
271  
-            job = self.jobs_all[num]
  292
+            job = self.all[num]
272 293
         except KeyError:
273 294
             error('Job #%s not found' % num)
274 295
         else:
@@ -277,12 +298,12 @@ def remove(self,num):
277 298
                 error('Job #%s is still running, it can not be removed.' % num)
278 299
                 return
279 300
             elif stat_code == self._s_completed:
280  
-                self.jobs_comp.remove(job)
  301
+                self.completed.remove(job)
281 302
             elif stat_code == self._s_dead:
282  
-                self.jobs_dead.remove(job)
  303
+                self.dead.remove(job)
283 304
 
284  
-    def flush_finished(self):
285  
-        """Flush all jobs finished (completed and dead) from lists.
  305
+    def flush(self):
  306
+        """Flush all finished jobs (completed and dead) from lists.
286 307
 
287 308
         Running jobs are never flushed.
288 309
 
@@ -290,35 +311,41 @@ def flush_finished(self):
290 311
         completed since the last _status_new() call, the flush operation
291 312
         aborts."""
292 313
 
293  
-        if self._status_new():
294  
-            error('New jobs completed since last '\
295  
-                  '_status_new(), aborting flush.')
296  
-            return
297  
-
298 314
         # Remove the finished jobs from the master dict
299  
-        jobs_all = self.jobs_all
300  
-        for job in self.jobs_comp+self.jobs_dead:
301  
-            del(jobs_all[job.num])
  315
+        alljobs = self.all
  316
+        for job in self.completed+self.dead:
  317
+            del(alljobs[job.num])
302 318
 
303 319
         # Now flush these lists completely
304  
-        fl_comp = self._group_flush(self.jobs_comp,'Completed')
305  
-        fl_dead = self._group_flush(self.jobs_dead,'Dead')
  320
+        fl_comp = self._group_flush(self.completed, 'Completed')
  321
+        fl_dead = self._group_flush(self.dead, 'Dead')
306 322
         if not (fl_comp or fl_dead):
307 323
             print 'No jobs to flush.'
308 324
 
309 325
     def result(self,num):
310 326
         """result(N) -> return the result of job N."""
311 327
         try:
312  
-            return self.jobs_all[num].result
  328
+            return self.all[num].result
313 329
         except KeyError:
314 330
             error('Job #%s not found' % num)
315 331
 
316  
-    def traceback(self,num):
  332
+    def _traceback(self, job):
  333
+        num = job if isinstance(job, int) else job.num
317 334
         try:
318  
-            self.jobs_all[num].traceback()
  335
+            self.all[num].traceback()
319 336
         except KeyError:
320 337
             error('Job #%s not found' % num)
321 338
 
  339
+    def traceback(self, job=None):
  340
+        if job is None:
  341
+            self._update_status()
  342
+            for deadjob in self.dead:
  343
+                print "Traceback for: %r" % deadjob
  344
+                self._traceback(deadjob)
  345
+                print
  346
+        else:
  347
+            self._traceback(job)
  348
+
322 349
 
323 350
 class BackgroundJobBase(threading.Thread):
324 351
     """Base class to build BackgroundJob classes.
@@ -360,14 +387,19 @@ def _init(self):
360 387
         self.stat_code = BackgroundJobBase.stat_created_c
361 388
         self.finished  = False
362 389
         self.result    = '<BackgroundJob has not completed>'
  390
+        
363 391
         # reuse the ipython traceback handler if we can get to it, otherwise
364 392
         # make a new one
365 393
         try:
366  
-            self._make_tb = __IPYTHON__.InteractiveTB.text
  394
+            make_tb = get_ipython().InteractiveTB.text
367 395
         except:
368  
-            self._make_tb = AutoFormattedTB(mode = 'Context',
369  
-                                           color_scheme='NoColor',
370  
-                                           tb_offset = 1).text
  396
+            make_tb = AutoFormattedTB(mode = 'Context',
  397
+                                      color_scheme='NoColor',
  398
+                                      tb_offset = 1).text
  399
+        # Note that the actual API for text() requires the three args to be
  400
+        # passed in, so we wrap it in a simple lambda.
  401
+        self._make_tb = lambda : make_tb(None, None, None)
  402
+
371 403
         # Hold a formatted traceback if one is generated.
372 404
         self._tb = None
373 405
         
@@ -377,7 +409,7 @@ def __str__(self):
377 409
         return self.strform
378 410
 
379 411
     def __repr__(self):
380  
-        return '<BackgroundJob: %s>' % self.strform
  412
+        return '<BackgroundJob #%d: %s>' % (self.num, self.strform)
381 413
 
382 414
     def traceback(self):
383 415
         print self._tb
@@ -398,10 +430,11 @@ def run(self):
398 430
             self.stat_code = BackgroundJobBase.stat_completed_c
399 431
             self.finished  = True
400 432
 
  433
+
401 434
 class BackgroundJobExpr(BackgroundJobBase):
402 435
     """Evaluate an expression as a background job (uses a separate thread)."""
403 436
 
404  
-    def __init__(self,expression,glob=None,loc=None):
  437
+    def __init__(self, expression, glob=None, loc=None):
405 438
         """Create a new job from a string which can be fed to eval().
406 439
 
407 440
         global/locals dicts can be provided, which will be passed to the eval
@@ -410,11 +443,8 @@ def __init__(self,expression,glob=None,loc=None):
410 443
         # fail immediately if the given expression can't be compiled
411 444
         self.code = compile(expression,'<BackgroundJob compilation>','eval')
412 445
                 
413  
-        if glob is None:
414  
-            glob = {}
415  
-        if loc is None:
416  
-            loc = {}
417  
-            
  446
+        glob = {} if glob is None else glob
  447
+        loc = {} if loc is None else loc
418 448
         self.expression = self.strform = expression
419 449
         self.glob = glob
420 450
         self.loc = loc
@@ -423,21 +453,19 @@ def __init__(self,expression,glob=None,loc=None):
423 453
     def call(self):
424 454
         return eval(self.code,self.glob,self.loc)
425 455
 
  456
+
426 457
 class BackgroundJobFunc(BackgroundJobBase):
427 458
     """Run a function call as a background job (uses a separate thread)."""
428 459
 
429  
-    def __init__(self,func,*args,**kwargs):
  460
+    def __init__(self, func, *args, **kwargs):
430 461
         """Create a new job from a callable object.
431 462
 
432 463
         Any positional arguments and keyword args given to this constructor
433 464
         after the initial callable are passed directly to it."""
434 465
 
435  
-        assert callable(func),'first argument must be callable'
436  
-        
437  
-        if args is None:
438  
-            args = []
439  
-        if kwargs is None:
440  
-            kwargs = {}
  466
+        if not callable(func):
  467
+            raise TypeError(
  468
+                'first argument to BackgroundJobFunc must be callable')
441 469
         
442 470
         self.func = func
443 471
         self.args = args
@@ -449,42 +477,4 @@ def __init__(self,func,*args,**kwargs):
449 477
         self._init()
450 478
 
451 479
     def call(self):
452  
-        return self.func(*self.args,**self.kwargs)
453  
-
454  
-
455  
-if __name__=='__main__':
456  
-
457  
-    import time
458  
-
459  
-    def sleepfunc(interval=2,*a,**kw):
460  
-        args = dict(interval=interval,
461  
-                    args=a,
462  
-                    kwargs=kw)
463  
-        time.sleep(interval)
464  
-        return args
465  
-
466  
-    def diefunc(interval=2,*a,**kw):
467  
-        time.sleep(interval)
468  
-        die
469  
-
470  
-    def printfunc(interval=1,reps=5):
471  
-        for n in range(reps):
472  
-            time.sleep(interval)
473  
-            print 'In the background...'
474  
-
475  
-    jobs = BackgroundJobManager()
476  
-    # first job will have # 0
477  
-    jobs.new(sleepfunc,4)
478  
-    jobs.new(sleepfunc,kw={'reps':2})
479  
-    # This makes a job which will die
480  
-    jobs.new(diefunc,1)
481  
-    jobs.new('printfunc(1,3)')
482  
-
483  
-    # after a while, you can get the traceback of a dead job.  Run the line
484  
-    # below again interactively until it prints a traceback (check the status
485  
-    # of the job):
486  
-    print jobs[1].status
487  
-    jobs[1].traceback()
488  
-    
489  
-    # Run this line again until the printed result changes
490  
-    print "The result of job #0 is:",jobs[0].result
  480
+        return self.func(*self.args, **self.kwargs)
91  IPython/lib/tests/test_backgroundjobs.py
... ...
@@ -0,0 +1,91 @@
  1
+"""Tests for pylab tools module.
  2
+"""
  3
+#-----------------------------------------------------------------------------
  4
+# Copyright (c) 2011, the IPython Development Team.
  5
+#
  6
+# Distributed under the terms of the Modified BSD License.
  7
+#
  8
+# The full license is in the file COPYING.txt, distributed with this software.
  9
+#-----------------------------------------------------------------------------
  10
+
  11
+#-----------------------------------------------------------------------------
  12
+# Imports
  13
+#-----------------------------------------------------------------------------
  14
+from __future__ import print_function
  15
+
  16
+# Stdlib imports
  17
+import sys
  18
+import time
  19
+
  20
+# Third-party imports
  21
+import nose.tools as nt
  22
+
  23
+# Our own imports
  24
+from IPython.lib import backgroundjobs as bg
  25
+from IPython.testing import decorators as dec
  26
+
  27
+#-----------------------------------------------------------------------------
  28
+# Globals and constants
  29
+#-----------------------------------------------------------------------------
  30
+t_short = 0.0001  # very short interval to wait on jobs
  31
+
  32
+#-----------------------------------------------------------------------------
  33
+# Local utilities
  34
+#-----------------------------------------------------------------------------
  35
+def sleeper(interval=t_short, *a, **kw):
  36
+    args = dict(interval=interval,
  37
+                other_args=a,
  38
+                kw_args=kw)
  39
+    time.sleep(interval)
  40
+    return args
  41
+
  42
+def crasher(interval=t_short, *a, **kw):
  43
+    time.sleep(interval)
  44
+    raise Exception("Dead job with interval %s" % interval)
  45
+
  46
+#-----------------------------------------------------------------------------
  47
+# Classes and functions
  48
+#-----------------------------------------------------------------------------
  49
+
  50
+def test_result():
  51
+    """Test job submission and result retrieval"""
  52
+    jobs = bg.BackgroundJobManager()
  53
+    j = jobs.new(sleeper)
  54
+    j.join()
  55
+    nt.assert_equals(j.result['interval'], t_short)
  56
+    
  57
+
  58
+def test_flush():
  59
+    """Test job control"""
  60
+    jobs = bg.BackgroundJobManager()
  61
+    j = jobs.new(sleeper)
  62
+    j.join()
  63
+    nt.assert_equals(len(jobs.completed), 1)
  64
+    nt.assert_equals(len(jobs.dead), 0)
  65
+    jobs.flush()
  66
+    nt.assert_equals(len(jobs.completed), 0)
  67
+    
  68
+
  69
+def test_dead():
  70
+    """Test control of dead jobs"""
  71
+    jobs = bg.BackgroundJobManager()
  72
+    j = jobs.new(crasher)
  73
+    j.join()
  74
+    nt.assert_equals(len(jobs.completed), 0)
  75
+    nt.assert_equals(len(jobs.dead), 1)
  76
+    jobs.flush()    
  77
+    nt.assert_equals(len(jobs.dead), 0)
  78
+
  79
+
  80
+def test_longer():
  81
+    """Test control of longer-running jobs"""
  82
+    jobs = bg.BackgroundJobManager()
  83
+    # Sleep for long enough for the following two checks to still report the
  84
+    # job as running, but not so long that it makes the test suite noticeably
  85
+    # slower. 
  86
+    j = jobs.new(sleeper, 0.1)
  87
+    nt.assert_equals(len(jobs.running), 1)
  88
+    nt.assert_equals(len(jobs.completed), 0)
  89
+    j.join()
  90
+    nt.assert_equals(len(jobs.running), 0)
  91
+    nt.assert_equals(len(jobs.completed), 1)
193  docs/examples/lib/BackgroundJobs.ipynb
... ...
@@ -0,0 +1,193 @@
  1
+{
  2
+    "worksheets": [
  3
+        {
  4
+            "cells": [
  5
+                {
  6
+                    "source": "# Simple interactive bacgkround jobs with IPython\n\nWe start by loading the `backgroundjobs` library and defining a few trivial functions to illustrate things with.", 
  7
+                    "cell_type": "markdown"
  8
+                }, 
  9
+                {
  10
+                    "cell_type": "code", 
  11
+                    "language": "python", 
  12
+                    "outputs": [], 
  13
+                    "collapsed": false, 
  14
+                    "prompt_number": 35, 
  15
+                    "input": "from IPython.lib import backgroundjobs as bg\n\nimport sys\nimport time\n\ndef sleepfunc(interval=2, *a, **kw):\n    args = dict(interval=interval,\n                args=a,\n                kwargs=kw)\n    time.sleep(interval)\n    return args\n\ndef diefunc(interval=2, *a, **kw):\n    time.sleep(interval)\n    raise Exception(\"Dead job with interval %s\" % interval)\n\ndef printfunc(interval=1, reps=5):\n    for n in range(reps):\n        time.sleep(interval)\n        print 'In the background...', n\n        sys.stdout.flush()\n    print 'All done!'\n    sys.stdout.flush()"
  16
+                }, 
  17
+                {
  18
+                    "source": "Now, we can create a job manager (called simply `jobs`) and use it to submit new jobs.\n<br>\nRun the cell below and wait a few seconds for the whole thing to finish, until you see the \"All done!\" printout.", 
  19
+                    "cell_type": "markdown"
  20
+                }, 
  21
+                {
  22
+                    "cell_type": "code", 
  23
+                    "language": "python", 
  24
+                    "outputs": [
  25
+                        {
  26
+                            "output_type": "stream", 
  27
+                            "stream": "stdout", 
  28
+                            "text": "Starting job # 0 in a separate thread.\nStarting job # 2 in a separate thread.\nStarting job # 3 in a separate thread.\nStarting job # 4 in a separate thread.\nStarting job # 5 in a separate thread.\n"
  29
+                        }, 
  30
+                        {
  31
+                            "output_type": "stream", 
  32
+                            "stream": "stdout", 
  33
+                            "text": "In the background... 0\n"
  34
+                        }, 
  35
+                        {
  36
+                            "output_type": "stream", 
  37
+                            "stream": "stdout", 
  38
+                            "text": "In the background... 1\n"
  39
+                        }, 
  40
+                        {
  41
+                            "output_type": "stream", 
  42
+                            "stream": "stdout", 
  43
+                            "text": "In the background... 2\n"
  44
+                        }, 
  45
+                        {
  46
+                            "output_type": "stream", 
  47
+                            "stream": "stdout", 
  48
+                            "text": "All done!\n"
  49
+                        }
  50
+                    ], 
  51
+                    "collapsed": false, 
  52
+                    "prompt_number": 36, 
  53
+                    "input": "jobs = bg.BackgroundJobManager()\n\n# Start a few jobs, the first one will have ID # 0\njobs.new(sleepfunc, 4)\njobs.new(sleepfunc, kw={'reps':2})\njobs.new('printfunc(1,3)')\n\n# This makes a couple of jobs which will die.  Let's keep a reference to\n# them for easier traceback reporting later\ndiejob1 = jobs.new(diefunc, 1)\ndiejob2 = jobs.new(diefunc, 2)"
  54
+                }, 
  55
+                {
  56
+                    "source": "You can check the status of your jobs at any time:", 
  57
+                    "cell_type": "markdown"
  58
+                }, 
  59
+                {
  60
+                    "cell_type": "code", 
  61
+                    "language": "python", 
  62
+                    "outputs": [
  63
+                        {
  64
+                            "output_type": "stream", 
  65
+                            "stream": "stdout", 
  66
+                            "text": "Completed jobs:\n0 : &lt;function sleepfunc at 0x30e1578&gt;\n2 : &lt;function sleepfunc at 0x30e1578&gt;\n3 : printfunc(1,3)\n\nDead jobs:\n4 : &lt;function diefunc at 0x304d488&gt;\n5 : &lt;function diefunc at 0x304d488&gt;\n\n"
  67
+                        }
  68
+                    ], 
  69
+                    "collapsed": false, 
  70
+                    "prompt_number": 37, 
  71
+                    "input": "jobs.status()"
  72
+                }, 
  73
+                {
  74
+                    "source": "For any completed job, you can get its result easily:", 
  75
+                    "cell_type": "markdown"
  76
+                }, 
  77
+                {
  78
+                    "cell_type": "code", 
  79
+                    "language": "python", 
  80
+                    "outputs": [], 
  81
+                    "collapsed": false, 
  82
+                    "prompt_number": 43, 
  83
+                    "input": "jobs[0].result\nj0 = jobs[0]\nj0.join?"
  84
+                }, 
  85
+                {
  86
+                    "source": "You can get the traceback of any dead job.  Run the line\nbelow again interactively until it prints a traceback (check the status\nof the job):\n", 
  87
+                    "cell_type": "markdown"
  88
+                }, 
  89
+                {
  90
+                    "cell_type": "code", 
  91
+                    "language": "python", 
  92
+                    "outputs": [
  93
+                        {
  94
+                            "output_type": "stream", 
  95
+                            "stream": "stdout", 
  96
+                            "text": "Status of diejob1: Dead (Exception), call jobs.traceback() for details\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span>                                 Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\">    462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">    463</span>     <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--&gt; 464</span><span class=\"ansiyellow\">         </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\">*</span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, *a, **kw)</span>\n<span class=\"ansigreen\">     14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     15</span>     time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---&gt; 16</span><span class=\"ansiyellow\">     </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">&quot;Dead job with interval %s&quot;</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n"
  97
+                        }
  98
+                    ], 
  99
+                    "collapsed": false, 
  100
+                    "prompt_number": 34, 
  101
+                    "input": "print \"Status of diejob1:\", diejob1.status\ndiejob1.traceback()  # jobs.traceback(4) would also work here, with the job number"
  102
+                }, 
  103
+                {
  104
+                    "source": "This will print all tracebacks for all dead jobs:", 
  105
+                    "cell_type": "markdown"
  106
+                }, 
  107
+                {
  108
+                    "cell_type": "code", 
  109
+                    "language": "python", 
  110
+                    "outputs": [
  111
+                        {
  112
+                            "output_type": "stream", 
  113
+                            "stream": "stdout", 
  114
+                            "text": "Traceback for: &lt;BackgroundJob #4: &lt;function diefunc at 0x30df758&gt;&gt;\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span>                                 Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\">    462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">    463</span>     <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--&gt; 464</span><span class=\"ansiyellow\">         </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\">*</span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, *a, **kw)</span>\n<span class=\"ansigreen\">     14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     15</span>     time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---&gt; 16</span><span class=\"ansiyellow\">     </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">&quot;Dead job with interval %s&quot;</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 1\n\nTraceback for: &lt;BackgroundJob #5: &lt;function diefunc at 0x30df758&gt;&gt;\n<span class=\"ansired\">---------------------------------------------------------------------------</span>\n<span class=\"ansired\">Exception</span>                                 Traceback (most recent call last)\n<span class=\"ansigreen\">/home/fperez/usr/lib/python2.6/site-packages/IPython/lib/backgroundjobs.py</span> in <span class=\"ansicyan\">call</span><span class=\"ansiblue\">(self)</span>\n<span class=\"ansigreen\">    462</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">    463</span>     <span class=\"ansigreen\">def</span> call<span class=\"ansiyellow\">(</span>self<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">--&gt; 464</span><span class=\"ansiyellow\">         </span><span class=\"ansigreen\">return</span> self<span class=\"ansiyellow\">.</span>func<span class=\"ansiyellow\">(</span><span class=\"ansiyellow\">*</span>self<span class=\"ansiyellow\">.</span>args<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>self<span class=\"ansiyellow\">.</span>kwargs<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansigreen\">/home/fperez/ipython/ipython/docs/examples/lib/&lt;ipython-input-15-54795a097787&gt;</span> in <span class=\"ansicyan\">diefunc</span><span class=\"ansiblue\">(interval, *a, **kw)</span>\n<span class=\"ansigreen\">     14</span> <span class=\"ansigreen\">def</span> diefunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">2</span><span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">*</span>a<span class=\"ansiyellow\">,</span> <span class=\"ansiyellow\">**</span>kw<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     15</span>     time<span class=\"ansiyellow\">.</span>sleep<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">---&gt; 16</span><span class=\"ansiyellow\">     </span><span class=\"ansigreen\">raise</span> Exception<span class=\"ansiyellow\">(</span><span class=\"ansiblue\">&quot;Dead job with interval %s&quot;</span> <span class=\"ansiyellow\">%</span> interval<span class=\"ansiyellow\">)</span><span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     17</span> <span class=\"ansiyellow\"></span>\n<span class=\"ansigreen\">     18</span> <span class=\"ansigreen\">def</span> printfunc<span class=\"ansiyellow\">(</span>interval<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">1</span><span class=\"ansiyellow\">,</span> reps<span class=\"ansiyellow\">=</span><span class=\"ansicyan\">5</span><span class=\"ansiyellow\">)</span><span class=\"ansiyellow\">:</span><span class=\"ansiyellow\"></span>\n\n<span class=\"ansired\">Exception</span>: Dead job with interval 2\n\n"
  115
+                        }
  116
+                    ], 
  117
+                    "collapsed": false, 
  118
+                    "prompt_number": 33, 
  119
+                    "input": "jobs.traceback()"
  120
+                }, 
  121
+                {
  122
+                    "source": "The job manager can be flushed of all completed jobs at any time:", 
  123
+                    "cell_type": "markdown"
  124
+                }, 
  125
+                {
  126
+                    "cell_type": "code", 
  127
+                    "language": "python", 
  128
+                    "outputs": [
  129
+                        {
  130
+                            "output_type": "stream", 
  131
+                            "stream": "stdout", 
  132
+                            "text": "No jobs to flush.\n"
  133
+                        }
  134
+                    ], 
  135
+                    "collapsed": false, 
  136
+                    "prompt_number": 25, 
  137
+                    "input": "jobs.flush()"
  138
+                }, 
  139
+                {
  140
+                    "source": "After that, the status is simply empty:", 
  141
+                    "cell_type": "markdown"
  142
+                }, 
  143
+                {
  144
+                    "cell_type": "code", 
  145
+                    "language": "python", 
  146
+                    "outputs": [], 
  147
+                    "collapsed": true, 
  148
+                    "prompt_number": 27, 
  149
+                    "input": "jobs.status()"
  150
+                }, 
  151
+                {
  152
+                    "source": "It's easy to wait on a job:", 
  153
+                    "cell_type": "markdown"
  154
+                }, 
  155
+                {
  156
+                    "cell_type": "code", 
  157
+                    "language": "python", 
  158
+                    "outputs": [
  159
+                        {
  160
+                            "output_type": "stream", 
  161
+                            "stream": "stdout", 
  162
+                            "text": "Starting job # 7 in a separate thread.\nWill wait for j now...\n"
  163
+                        }, 
  164
+                        {
  165
+                            "output_type": "stream", 
  166
+                            "stream": "stdout", 
  167
+                            "text": "Result from j:\n"
  168
+                        }, 
  169
+                        {
  170
+                            "output_type": "pyout", 
  171
+                            "prompt_number": 46, 
  172
+                            "text": "{&apos;args&apos;: (), &apos;interval&apos;: 2, &apos;kwargs&apos;: {}}"
  173
+                        }
  174
+                    ], 
  175
+                    "collapsed": false, 
  176
+                    "prompt_number": 46, 
  177
+                    "input": "j = jobs.new(sleepfunc, 2)\nprint \"Will wait for j now...\"\nsys.stdout.flush()\nj.join()\nprint \"Result from j:\"\nj.result"
  178
+                }, 
  179
+                {
  180
+                    "input": "", 
  181
+                    "cell_type": "code", 
  182
+                    "collapsed": true, 
  183
+                    "language": "python", 
  184
+                    "outputs": []
  185
+                }
  186
+            ]
  187
+        }
  188
+    ], 
  189
+    "metadata": {
  190
+        "name": "BackgroundJobs"
  191
+    }, 
  192
+    "nbformat": 2
  193
+}
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.