/gobyexample

Permalink
Find file
gobyexample/public/worker-pools
Fetching contributors…
Cannot retrieve contributors at this time
Raw Blame History
217 lines (166 sloc) 8.88 KB
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Go by Example: Worker Pools</title>
<link rel=stylesheet href="site.css">
</head>
<script type="text/javascript">
if (window.location.host == "gobyexample.com") {
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-34996217-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
}
</script>
<body>
<div class="example" id="worker-pools">
<h2><a href="./">Go by Example</a>: Worker Pools</h2>
<table>
<tr>
<td class="docs">
<p>In this example we&rsquo;ll look at how to implement
a <em>worker pool</em> using goroutines and channels.</p>
</td>
<td class="code empty leading">
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<a href="http://play.golang.org/p/y12QxBsKtT"><img title="Run code" src="play.png" class="run" /></a>
<div class="highlight"><pre><span class="kn">package</span> <span class="nx">main</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kn">import</span> <span class="s">&quot;fmt&quot;</span>
<span class="kn">import</span> <span class="s">&quot;time&quot;</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Here&rsquo;s the worker, of which we&rsquo;ll run several
concurrent instances. These workers will receive
work on the <code>jobs</code> channel and send the corresponding
results on <code>results</code>. We&rsquo;ll sleep a second per job to
simulate an expensive task.</p>
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">id</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">jobs</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">results</span> <span class="kd">chan</span><span class="o">&lt;-</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">jobs</span> <span class="p">{</span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;worker&quot;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="s">&quot;processing job&quot;</span><span class="p">,</span> <span class="nx">j</span><span class="p">)</span>
<span class="nx">time</span><span class="p">.</span><span class="nx">Sleep</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
<span class="nx">results</span> <span class="o">&lt;-</span> <span class="nx">j</span> <span class="o">*</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>In order to use our pool of workers we need to send
them work and collect their results. We make 2
channels for this.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="nx">jobs</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
<span class="nx">results</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>This starts up 3 workers, initially blocked
because there are no jobs yet.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">w</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">w</span> <span class="o">&lt;=</span> <span class="mi">3</span><span class="p">;</span> <span class="nx">w</span><span class="o">++</span> <span class="p">{</span>
<span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="nx">jobs</span><span class="p">,</span> <span class="nx">results</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Here we send 9 <code>jobs</code> and then <code>close</code> that
channel to indicate that&rsquo;s all the work we have.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">j</span> <span class="o">&lt;=</span> <span class="mi">9</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span> <span class="p">{</span>
<span class="nx">jobs</span> <span class="o">&lt;-</span> <span class="nx">j</span>
<span class="p">}</span>
<span class="nb">close</span><span class="p">(</span><span class="nx">jobs</span><span class="p">)</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Finally we collect all the results of the work.</p>
</td>
<td class="code">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">a</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">a</span> <span class="o">&lt;=</span> <span class="mi">9</span><span class="p">;</span> <span class="nx">a</span><span class="o">++</span> <span class="p">{</span>
<span class="o">&lt;-</span><span class="nx">results</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
</table>
<table>
<tr>
<td class="docs">
<p>Our running program shows the 9 jobs being executed by
various workers. The program only takes about 3 seconds
despite doing about 9 seconds of total work because
there are 3 workers operating concurrently.</p>
</td>
<td class="code leading">
<div class="highlight"><pre><span class="gp">$</span> <span class="nb">time </span>go run worker-pools.go
<span class="go">worker 1 processing job 1</span>
<span class="go">worker 2 processing job 2</span>
<span class="go">worker 3 processing job 3</span>
<span class="go">worker 1 processing job 4</span>
<span class="go">worker 2 processing job 5</span>
<span class="go">worker 3 processing job 6</span>
<span class="go">worker 1 processing job 7</span>
<span class="go">worker 2 processing job 8</span>
<span class="go">worker 3 processing job 9</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code">
<div class="highlight"><pre><span class="go">real 0m3.149s</span>
</pre></div>
</td>
</tr>
</table>
<p class="next">
Next example: <a href="rate-limiting">Rate Limiting</a>.
</p>
<p class="footer">
by <a href="https://twitter.com/mmcgrana">@mmcgrana</a> | <a href="mailto:mmcgrana@gmail.com">feedback</a> | <a href="https://github.com/mmcgrana/gobyexample/blob/master/examples/worker-pools">source</a> | <a href="https://github.com/mmcgrana/gobyexample#license">license</a>
</p>
</div>
</body>
</html>