Skip to content

Commit

Permalink
Fix signature description broken in PySpark API documentation in 2.2.2
Browse files Browse the repository at this point in the history
  • Loading branch information
HyukjinKwon committed Jul 9, 2018
1 parent 2b5ba2f commit 7b3e459
Show file tree
Hide file tree
Showing 6 changed files with 238 additions and 237 deletions.
22 changes: 11 additions & 11 deletions site/docs/2.2.2/api/python/pyspark.html
Expand Up @@ -264,7 +264,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<li>in case of YARN something like ‘application_1433865536131_34483’</li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sc</span><span class="o">.</span><span class="n">applicationId</span>
<span class="go">u&#39;local-...&#39;</span>
<span class="go">&#39;local-...&#39;</span>
</pre></div>
</div>
</dd></dl>
Expand Down Expand Up @@ -743,7 +743,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<span class="gp">... </span> <span class="n">_</span> <span class="o">=</span> <span class="n">testFile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&quot;Hello world!&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">textFile</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">textFile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">textFile</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="go">[u&#39;Hello world!&#39;]</span>
<span class="go">[&#39;Hello world!&#39;]</span>
</pre></div>
</div>
</dd></dl>
Expand All @@ -766,10 +766,10 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<span class="gp">... </span> <span class="n">_</span> <span class="o">=</span> <span class="n">testFile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&quot;Hello&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">textFile</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">textFile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">textFile</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="go">[u&#39;Hello&#39;]</span>
<span class="go">[&#39;Hello&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">parallelized</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="s2">&quot;World!&quot;</span><span class="p">])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">sorted</span><span class="p">(</span><span class="n">sc</span><span class="o">.</span><span class="n">union</span><span class="p">([</span><span class="n">textFile</span><span class="p">,</span> <span class="n">parallelized</span><span class="p">])</span><span class="o">.</span><span class="n">collect</span><span class="p">())</span>
<span class="go">[u&#39;Hello&#39;, &#39;World!&#39;]</span>
<span class="go">[&#39;Hello&#39;, &#39;World!&#39;]</span>
</pre></div>
</div>
</dd></dl>
Expand Down Expand Up @@ -819,7 +819,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<span class="gp">... </span> <span class="n">_</span> <span class="o">=</span> <span class="n">file2</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&quot;2&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">textFiles</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">wholeTextFiles</span><span class="p">(</span><span class="n">dirPath</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">sorted</span><span class="p">(</span><span class="n">textFiles</span><span class="o">.</span><span class="n">collect</span><span class="p">())</span>
<span class="go">[(u&#39;.../1.txt&#39;, u&#39;1&#39;), (u&#39;.../2.txt&#39;, u&#39;2&#39;)]</span>
<span class="go">[(&#39;.../1.txt&#39;, &#39;1&#39;), (&#39;.../2.txt&#39;, &#39;2&#39;)]</span>
</pre></div>
</div>
</dd></dl>
Expand Down Expand Up @@ -1684,7 +1684,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<code class="descname">pipe</code><span class="sig-paren">(</span><em>command</em>, <em>env=None</em>, <em>checkCode=False</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/rdd.html#RDD.pipe"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.RDD.pipe" title="Permalink to this definition"></a></dt>
<dd><p>Return an RDD created by piping elements to a forked external process.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="s1">&#39;1&#39;</span><span class="p">,</span> <span class="s1">&#39;2&#39;</span><span class="p">,</span> <span class="s1">&#39;&#39;</span><span class="p">,</span> <span class="s1">&#39;3&#39;</span><span class="p">])</span><span class="o">.</span><span class="n">pipe</span><span class="p">(</span><span class="s1">&#39;cat&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="go">[u&#39;1&#39;, u&#39;2&#39;, u&#39;&#39;, u&#39;3&#39;]</span>
<span class="go">[&#39;1&#39;, &#39;2&#39;, &#39;&#39;, &#39;3&#39;]</span>
</pre></div>
</div>
<table class="docutils field-list" frame="void" rules="none">
Expand Down Expand Up @@ -1799,7 +1799,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi

<dl class="method">
<dt id="pyspark.RDD.repartitionAndSortWithinPartitions">
<code class="descname">repartitionAndSortWithinPartitions</code><span class="sig-paren">(</span><em>numPartitions=None</em>, <em>partitionFunc=&lt;function portable_hash&gt;</em>, <em>ascending=True</em>, <em>keyfunc=&lt;function &lt;lambda&gt;&gt;</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/rdd.html#RDD.repartitionAndSortWithinPartitions"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.RDD.repartitionAndSortWithinPartitions" title="Permalink to this definition"></a></dt>
<code class="descname">repartitionAndSortWithinPartitions</code><span class="sig-paren">(</span><em>numPartitions=None</em>, <em>partitionFunc=&lt;function portable_hash&gt;</em>, <em>ascending=True</em>, <em>keyfunc=&lt;function RDD.&lt;lambda&gt;&gt;</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/rdd.html#RDD.repartitionAndSortWithinPartitions"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.RDD.repartitionAndSortWithinPartitions" title="Permalink to this definition"></a></dt>
<dd><p>Repartition the RDD according to the given partitioner and, within each resulting partition,
sort records by their keys.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">rdd</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">8</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">6</span><span class="p">),</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">),</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">8</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">)])</span>
Expand Down Expand Up @@ -2089,7 +2089,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">fileinput</span> <span class="k">import</span> <span class="nb">input</span><span class="p">,</span> <span class="n">hook_compressed</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="n">glob</span><span class="p">(</span><span class="n">tempFile3</span><span class="o">.</span><span class="n">name</span> <span class="o">+</span> <span class="s2">&quot;/part*.gz&quot;</span><span class="p">),</span> <span class="n">openhook</span><span class="o">=</span><span class="n">hook_compressed</span><span class="p">))</span>
<span class="gp">&gt;&gt;&gt; </span><span class="sa">b</span><span class="s1">&#39;&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">result</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="go">u&#39;bar\nfoo\n&#39;</span>
<span class="go">&#39;bar\nfoo\n&#39;</span>
</pre></div>
</div>
</dd></dl>
Expand All @@ -2100,7 +2100,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi
<dd><p>Assign a name to this RDD.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">rdd1</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">rdd1</span><span class="o">.</span><span class="n">setName</span><span class="p">(</span><span class="s1">&#39;RDD1&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">name</span><span class="p">()</span>
<span class="go">u&#39;RDD1&#39;</span>
<span class="go">&#39;RDD1&#39;</span>
</pre></div>
</div>
</dd></dl>
Expand All @@ -2120,7 +2120,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi

<dl class="method">
<dt id="pyspark.RDD.sortByKey">
<code class="descname">sortByKey</code><span class="sig-paren">(</span><em>ascending=True</em>, <em>numPartitions=None</em>, <em>keyfunc=&lt;function &lt;lambda&gt;&gt;</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/rdd.html#RDD.sortByKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.RDD.sortByKey" title="Permalink to this definition"></a></dt>
<code class="descname">sortByKey</code><span class="sig-paren">(</span><em>ascending=True</em>, <em>numPartitions=None</em>, <em>keyfunc=&lt;function RDD.&lt;lambda&gt;&gt;</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/rdd.html#RDD.sortByKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.RDD.sortByKey" title="Permalink to this definition"></a></dt>
<dd><p>Sorts this RDD, which is assumed to consist of (key, value) pairs.
# noqa</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">tmp</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;1&#39;</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;d&#39;</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;2&#39;</span><span class="p">,</span> <span class="mi">5</span><span class="p">)]</span>
Expand Down Expand Up @@ -2664,7 +2664,7 @@ <h2>Subpackages<a class="headerlink" href="#subpackages" title="Permalink to thi

<dl class="method">
<dt id="pyspark.PickleSerializer.loads">
<code class="descname">loads</code><span class="sig-paren">(</span><em>obj</em>, <em>encoding=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/serializers.html#PickleSerializer.loads"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.PickleSerializer.loads" title="Permalink to this definition"></a></dt>
<code class="descname">loads</code><span class="sig-paren">(</span><em>obj</em>, <em>encoding='bytes'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/serializers.html#PickleSerializer.loads"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.PickleSerializer.loads" title="Permalink to this definition"></a></dt>
<dd><p>Deserialize an object from a byte array.</p>
</dd></dl>

Expand Down

0 comments on commit 7b3e459

Please sign in to comment.