Skip to content

Improved Error Handling #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 14, 2018
Merged

Improved Error Handling #246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
313f5cf
sync w/ internal
leewyang Mar 13, 2018
4390e22
bump version & docs
leewyang Mar 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/.buildinfo
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: 874109c9e8f56215fdcb46cac4aab9f9
config: abbb35398bf3c41c0f421213a6263bf9
tags: 645f666f9bcd5a90fca523b33c5a78b7
35 changes: 12 additions & 23 deletions docs/_modules/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,18 @@

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Overview: module code &#8212; TensorFlowOnSpark 1.2.0 documentation</title>
<title>Overview: module code &#8212; TensorFlowOnSpark 1.2.1 documentation</title>
<link rel="stylesheet" href="../_static/classic.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '1.2.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/documentation_options.js"></script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
</head>
<body>
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
Expand All @@ -34,7 +25,7 @@ <h3>Navigation</h3>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">TensorFlowOnSpark 1.2.0 documentation</a> &#187;</li>
<li class="nav-item nav-item-0"><a href="../index.html">TensorFlowOnSpark 1.2.1 documentation</a> &#187;</li>
</ul>
</div>

Expand All @@ -44,15 +35,11 @@ <h3>Navigation</h3>
<div class="body" role="main">

<h1>All modules for which code is available</h1>
<ul><li><a href="pyspark/ml/param.html">pyspark.ml.param</a></li>
<li><a href="tensorflowonspark/TFCluster.html">tensorflowonspark.TFCluster</a></li>
<li><a href="tensorflowonspark/TFManager.html">tensorflowonspark.TFManager</a></li>
<ul><li><a href="tensorflowonspark/TFManager.html">tensorflowonspark.TFManager</a></li>
<li><a href="tensorflowonspark/TFNode.html">tensorflowonspark.TFNode</a></li>
<li><a href="tensorflowonspark/TFSparkNode.html">tensorflowonspark.TFSparkNode</a></li>
<li><a href="tensorflowonspark/dfutil.html">tensorflowonspark.dfutil</a></li>
<li><a href="tensorflowonspark/gpu_info.html">tensorflowonspark.gpu_info</a></li>
<li><a href="tensorflowonspark/marker.html">tensorflowonspark.marker</a></li>
<li><a href="tensorflowonspark/pipeline.html">tensorflowonspark.pipeline</a></li>
<li><a href="tensorflowonspark/reservation.html">tensorflowonspark.reservation</a></li>
<li><a href="tensorflowonspark/util.html">tensorflowonspark.util</a></li>
</ul>
Expand All @@ -64,12 +51,14 @@ <h1>All modules for which code is available</h1>
<div class="sphinxsidebarwrapper">
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
Expand All @@ -85,12 +74,12 @@ <h3>Navigation</h3>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">TensorFlowOnSpark 1.2.0 documentation</a> &#187;</li>
<li class="nav-item nav-item-0"><a href="../index.html">TensorFlowOnSpark 1.2.1 documentation</a> &#187;</li>
</ul>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright 2017, Yahoo Inc.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.1.
</div>
</body>
</html>
50 changes: 25 additions & 25 deletions docs/_modules/tensorflowonspark/TFManager.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,18 @@

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>tensorflowonspark.TFManager &#8212; TensorFlowOnSpark 1.2.0 documentation</title>
<title>tensorflowonspark.TFManager &#8212; TensorFlowOnSpark 1.2.1 documentation</title>
<link rel="stylesheet" href="../../_static/classic.css" type="text/css" />
<link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../../',
VERSION: '1.2.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../_static/documentation_options.js"></script>
<script type="text/javascript" src="../../_static/jquery.js"></script>
<script type="text/javascript" src="../../_static/underscore.js"></script>
<script type="text/javascript" src="../../_static/doctools.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
</head>
<body>
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
Expand All @@ -34,7 +25,7 @@ <h3>Navigation</h3>
<li class="right" >
<a href="../../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="nav-item nav-item-0"><a href="../../index.html">TensorFlowOnSpark 1.2.0 documentation</a> &#187;</li>
<li class="nav-item nav-item-0"><a href="../../index.html">TensorFlowOnSpark 1.2.1 documentation</a> &#187;</li>
<li class="nav-item nav-item-1"><a href="../index.html" accesskey="U">Module code</a> &#187;</li>
</ul>
</div>
Expand All @@ -58,8 +49,8 @@ <h1>Source code for tensorflowonspark.TFManager</h1><div class="highlight"><pre>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">JoinableQueue</span>

<div class="viewcode-block" id="TFManager"><a class="viewcode-back" href="../../tensorflowonspark.TFManager.html#tensorflowonspark.TFManager.TFManager">[docs]</a><span class="k">class</span> <span class="nc">TFManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;Python multiprocessing.Manager for distributed, multi-process communication.&quot;&quot;&quot;</span>
<span class="k">pass</span></div>
<span class="sd">&quot;&quot;&quot;Python multiprocessing.Manager for distributed, multi-process communication.&quot;&quot;&quot;</span></div>
<span class="k">pass</span>


<span class="c1"># global to each Spark executor&#39;s python worker</span>
Expand All @@ -73,6 +64,12 @@ <h1>Source code for tensorflowonspark.TFManager</h1><div class="highlight"><pre>
<span class="k">def</span> <span class="nf">_set</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="n">kdict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>

<span class="k">def</span> <span class="nf">_get_queue</span><span class="p">(</span><span class="n">qname</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="n">qdict</span><span class="p">[</span><span class="n">qname</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>

<div class="viewcode-block" id="start"><a class="viewcode-back" href="../../tensorflowonspark.TFManager.html#tensorflowonspark.TFManager.start">[docs]</a><span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="n">authkey</span><span class="p">,</span> <span class="n">queues</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">&#39;local&#39;</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;Create a new multiprocess.Manager (or return existing one).</span>

Expand All @@ -89,15 +86,16 @@ <h1>Source code for tensorflowonspark.TFManager</h1><div class="highlight"><pre>
<span class="n">kdict</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
<span class="k">for</span> <span class="n">q</span> <span class="ow">in</span> <span class="n">queues</span><span class="p">:</span>
<span class="n">qdict</span><span class="p">[</span><span class="n">q</span><span class="p">]</span> <span class="o">=</span> <span class="n">JoinableQueue</span><span class="p">()</span>
<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;get_queue&#39;</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span> <span class="n">qname</span><span class="p">:</span> <span class="n">qdict</span><span class="p">[</span><span class="n">qname</span><span class="p">])</span>

<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;get_queue&#39;</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span> <span class="n">qname</span><span class="p">:</span> <span class="n">_get_queue</span><span class="p">(</span><span class="n">qname</span><span class="p">))</span>
<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;get&#39;</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span> <span class="n">key</span><span class="p">:</span> <span class="n">_get</span><span class="p">(</span><span class="n">key</span><span class="p">))</span>
<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;set&#39;</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="n">_set</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
<span class="k">if</span> <span class="n">mode</span> <span class="o">==</span> <span class="s1">&#39;remote&#39;</span><span class="p">:</span>
<span class="n">mgr</span> <span class="o">=</span> <span class="n">TFManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">&#39;&#39;</span><span class="p">,</span><span class="mi">0</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="n">authkey</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">mgr</span> <span class="o">=</span> <span class="n">TFManager</span><span class="p">(</span><span class="n">authkey</span><span class="o">=</span><span class="n">authkey</span><span class="p">)</span>
<span class="n">mgr</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">return</span> <span class="n">mgr</span></div>
<span class="n">mgr</span><span class="o">.</span><span class="n">start</span><span class="p">()</span></div>
<span class="k">return</span> <span class="n">mgr</span>

<div class="viewcode-block" id="connect"><a class="viewcode-back" href="../../tensorflowonspark.TFManager.html#tensorflowonspark.TFManager.connect">[docs]</a><span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">authkey</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;Connect to a multiprocess.Manager.</span>
Expand All @@ -113,8 +111,8 @@ <h1>Source code for tensorflowonspark.TFManager</h1><div class="highlight"><pre>
<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;get&#39;</span><span class="p">)</span>
<span class="n">TFManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">&#39;set&#39;</span><span class="p">)</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">TFManager</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">authkey</span><span class="o">=</span><span class="n">authkey</span><span class="p">)</span>
<span class="n">m</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
<span class="k">return</span> <span class="n">m</span></div>
<span class="n">m</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span></div>
<span class="k">return</span> <span class="n">m</span>

</pre></div>

Expand All @@ -125,12 +123,14 @@ <h1>Source code for tensorflowonspark.TFManager</h1><div class="highlight"><pre>
<div class="sphinxsidebarwrapper">
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="../../search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
Expand All @@ -146,13 +146,13 @@ <h3>Navigation</h3>
<li class="right" >
<a href="../../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="nav-item nav-item-0"><a href="../../index.html">TensorFlowOnSpark 1.2.0 documentation</a> &#187;</li>
<li class="nav-item nav-item-0"><a href="../../index.html">TensorFlowOnSpark 1.2.1 documentation</a> &#187;</li>
<li class="nav-item nav-item-1"><a href="../index.html" >Module code</a> &#187;</li>
</ul>
</div>
<div class="footer" role="contentinfo">
&#169; Copyright 2017, Yahoo Inc.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.1.
</div>
</body>
</html>
Loading