Merge pull request #1795 from davidmarin/v0.6.3-release

v0.6.3 release

CHANGES.txt

@@ -1,3 +1,41 @@
+v0.6.3, 2018-05-31 -- Dataproc parity
+ * jobs:
+   * use mapper_raw() to read entire file, in any format (#754)
+ * log interpretation:
+   * handles "not a valid JAR" error from Hadoop (#1771)
+ * less dependencies on Google libraries (#1746):
+   * google-cloud-logging 1.5.0+
+   * google-cloud-storage 1.9.0+
+   * google-cloud-dataproc is vendored (future releases will require 0.11.0+)
+ * RetryWrapper now sets __name__ of wrapped functions (#1790)
+ * runners:
+   * all runners:
+     * don't stream output if --output-dir is specified (#1739)
+       * --no-output switch is now --no-cat-output
+       * added --cat-output switch
+   * cloud runners (Dataproc, EMR):
+     * renamed cloud_upload_part_size option to cloud_part_size_mb (#1774)
+   * DataprocJobRunner:
+     * options now supported:
+       * cloud_part_size_mb: control chunked uploading (#1404)
+       * {core,master,task}_instance_config (#1681):
+         * set disk_config, is_preemptible, other instance options
+       * cluster_config: set properties in Hadoop config files (#1680)
+       * hadoop_streaming_jar: specify custom Hadoop streaming JAR (#1676)
+       * network/subnet: specify network/subnetwork (#1683)
+       * service_account: specify custom IAM service account (#1682)
+       * service_account_scopes: specify custom permissions for cluster (#1682)
+       * ssh_tunnel/ssh_tunnel_is_open: access resource manager (#1670)
+     * fs:
+       * cat() streams data rather than dumping to a temp file (#1674)
+       * exists() no longer swallows exceptions (#1675)
+     * full support for parsing probable cause of job failure (#1672)
+     * full support for fetching and parsing counters (#1703)
+     * job progress messages (#1671)
+     * can now run JAR steps (#1677)
+     * uses Dataproc's built-in idle timeout, not a script (#1705)
+     * bootstrap script runs in temp dir, not / (#1601)
+
 v0.6.2, 2018-03-23 -- log parsing at scale
  * runners:
    * local runners

README.rst

@@ -6,7 +6,7 @@ mrjob: the Python MapReduce library
 mrjob is a Python 2.7/3.3+ package that helps you write and run Hadoop
 Streaming jobs.
 
-`Stable version (v0.6.2) documentation <http://mrjob.readthedocs.org/en/stable/>`_
+`Stable version (v0.6.3) documentation <http://mrjob.readthedocs.org/en/stable/>`_
 
 `Development version documentation <http://mrjob.readthedocs.org/en/latest/>`_
 

docs/_templates/indexsidebar.html

@@ -3,8 +3,8 @@ <h3>Quick Links</h3>
   <li><a href="{{ pathto('guides/quickstart') }}">Fundamentals</a></li>
   <li><a href="{{ pathto('guides/writing-mrjobs') }}">Writing jobs</a></li>
   <li><a href="{{ pathto('guides/runners') }}">Runners</a></li>
-  <li><a href="{{ pathto('guides/emr') }}">Elastic MapReduce</a></li>
-  <li><a href="{{ pathto('guides/dataproc') }}">Cloud Dataproc</a></li>
+  <li><a href="{{ pathto('guides/emr') }}">Amazon Elastic MapReduce</a></li>
+  <li><a href="{{ pathto('guides/dataproc') }}">Google Cloud Dataproc</a></li>
 </ul>
 <ul class="quick-links">
   <li><a href="{{ pathto('guides/configs-reference') }}">
@@ -14,7 +14,11 @@ <h3>Quick Links</h3>
   <li><a href="{{ pathto('guides/configs-hadoopy-runners') }}">
     Config options (Hadoop)</a></li>
   <li><a href="{{ pathto('guides/emr-opts') }}">
-    Config options (EMR)</a></li>
-  <li><a href="{{ pathto('guides/dataproc-opts') }}">
-    Config options (Dataproc)</a></li>
+      Config options (cloud services)</a></li>
+  <ul>
+    <li><a href="{{ pathto('guides/emr-opts') }}">
+        Amazon Elastic MapReduce</a></li>
+    <li><a href="{{ pathto('guides/dataproc-opts') }}">
+        Google Cloud Dataproc</a></li>
+  </ul>
 </ul>

docs/conf.py

@@ -34,7 +34,7 @@
 # project info
 
 project = u'mrjob'
-copyright = u'2009-2016 Yelp and Contributors'
+copyright = u'2009-2018 Yelp and Contributors'
 # The short X.Y version. Can refer to in docs with |version|.
 version = mrjob.__version__.split('-')[0]
 # The full version, including alpha/beta/rc tags.

docs/guides.rst

@@ -14,6 +14,7 @@ Guides
     guides/configs-all-runners.rst
     guides/configs-hadoopy-runners.rst
     guides/configs-reference.rst
+    guides/cloud-opts.rst
     guides/setup-cookbook.rst
     guides/hadoop-cookbook.rst
     guides/testing.rst