Merge branch 'zwei' into trains

Author: laurenyu

doc/api/utility/image_uris.rst

@@ -0,0 +1,7 @@
+Image URIs
+----------
+
+.. automodule:: sagemaker.image_uris
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/frameworks/tensorflow/upgrade_from_legacy.rst

@@ -248,7 +248,6 @@ For example, if you want to use JSON serialization and deserialization:
     from sagemaker.deserializers import JSONDeserializer
     from sagemaker.serializers import JSONSerializer
 
-    predictor.serializer = JSONSerializer()
-    predictor.deserializer = JSONDeserializer()
+    predictor = model.deploy(..., serializer=JSONSerializer(), deserializer=JSONDeserializer())
 
     predictor.predict(data)

doc/frameworks/xgboost/using_xgboost.rst

@@ -192,12 +192,14 @@ inference against your model.
 
 .. code::
 
+    serializer = StringSerializer()
+    serializer.CONTENT_TYPE = "text/libsvm"
+
     predictor = estimator.deploy(
         initial_instance_count=1,
-        instance_type="ml.m5.xlarge"
+        instance_type="ml.m5.xlarge",
+        serializer=serializer
     )
-    predictor.serializer = str
-    predictor.content_type = "text/libsvm"
 
     with open("abalone") as f:
         payload = f.read()

doc/index.rst

@@ -1,7 +1,3 @@
-.. important::
-    We are working on v2.0.0. See https://github.com/aws/sagemaker-python-sdk/issues/1459
-    for more info on our plans and to leave feedback!
-
 ###########################
 Amazon SageMaker Python SDK
 ###########################

doc/v2.rst

@@ -1,42 +1,29 @@
-##########################################################
-Use Version 2.0 of the SageMaker Python SDK (experimental)
-##########################################################
-
-Development on v2.0.0 of the SageMaker Python SDK is underway.
-For more info on our plans, see https://github.com/aws/sagemaker-python-sdk/issues/1459.
+###########################################
+Use Version 2.x of the SageMaker Python SDK
+###########################################
 
 .. contents::
     :local:
+    :depth: 2
 
 ************
 Installation
 ************
 
-.. warning::
-    Version 2.0.0 is currently experimental, so proceed with caution. If you do run into issues or have any other feedback,
-    please let us know by `opening an issue <https://github.com/aws/sagemaker-python-sdk/issues/new/choose>`_
-    or `commenting on our planning issue <https://github.com/aws/sagemaker-python-sdk/issues/1459>`_.
-
-To install the latest release candidate:
-
-.. code:: bash
-
-    pip install git+git://github.com/aws/sagemaker-python-sdk.git@v2.0.0.rc1
-
-To install the latest version of v2:
+To install the latest version:
 
 .. code:: bash
 
-    pip install git+git://github.com/aws/sagemaker-python-sdk.git@zwei
+    pip install --upgrade sagemaker
 
 If you are executing this pip install command in a notebook, make sure to restart your kernel.
 
-*******
-Changes
-*******
+****************
+Breaking Changes
+****************
 
 This section is for major changes that may require updates to your SageMaker Python SDK code.
-You can also see what changes have been made in the `CHANGELOG <https://github.com/aws/sagemaker-python-sdk/blob/zwei/CHANGELOG.md>`_.
+For the full list of changes, see the `CHANGELOG <https://github.com/aws/sagemaker-python-sdk/blob/master/CHANGELOG.md>`_.
 
 Deprecations
 ============
@@ -94,6 +81,13 @@ Please instantiate the objects instead.
 The ``update_endpoint`` argument in ``deploy()`` methods for estimators and models has been deprecated.
 Please use :func:`sagemaker.predictor.Predictor.update_endpoint` instead.
 
+``serializer`` and ``deserializer`` in ``create_model()``
+---------------------------------------------------------
+
+The ``serializer`` and ``deserializer`` arguments in
+:func:`sagemaker.estimator.Estimator.create_model` have been deprecated. Please
+specify serializers and deserializers in ``deploy()`` methods instead.
+
 ``content_type`` and ``accept`` in the Predictor Constructor
 ------------------------------------------------------------
 
@@ -127,8 +121,46 @@ write MIME types as a string,
 | ``CONTENT_TYPE_NPY``          | ``"application/x-npy"``        |
 +-------------------------------+--------------------------------+
 
+Image URI Functions (e.g. ``get_image_uri``)
+--------------------------------------------
+
+The following functions have been deprecated in favor of :func:`sagemaker.image_uris.retrieve`:
+
+- ``sagemaker.amazon_estimator.get_image_uri()``
+- ``sagemaker.fw_utils.create_image_uri()``
+- ``sagemaker.fw_registry.registry()``
+- ``sagemaker.utils.get_ecr_image_uri_prefix()``
+
+For more information about usage, see :func:`sagemaker.image_uris.retrieve`.
+
+SageMaker Python SDK CLI
+------------------------
+
+The SageMaker Python SDK CLI has been deprecated.
+(This is different from the AWS CLI.)
+
+``enable_cloudwatch_metrics`` for Estimators and Models
+-------------------------------------------------------
+
+The parameter ``enable_cloudwatch_metrics`` has been deprecated.
+CloudWatch metrics are already emitted for all Training Jobs, etc.
+
+``sagemaker.fw_utils.parse_s3_url``
+-----------------------------------
+
+The ``sagemaker.fw_utils.parse_s3_url`` function has been deprecated.
+Please use :func:`sagemaker.s3.parse_s3_url` instead.
+
+``sagemaker.session.ModelContainer``
+------------------------------------
+
+The class ``sagemaker.session.ModelContainer`` has been deprecated, as it is not needed for creating inference pipelines.
+
+Changes in Default Behavior
+===========================
+
 Require ``framework_version`` and ``py_version`` for Frameworks
-===============================================================
+---------------------------------------------------------------
 
 Framework estimator and model classes now require ``framework_version`` and ``py_version`` instead of supplying defaults,
 unless an image URI is explicitly supplied.
@@ -158,6 +190,25 @@ For example:
         py_version="py3",  # now required
     )
 
+Log Display Behavior with ``attach()``
+--------------------------------------
+
+Logs are no longer printed when using ``attach()`` with an estimator.
+To view logs after attaching a training job to an estimator, use :func:`sagemaker.estimator.EstimatorBase.logs`.
+
+``HyperparameterTuner.fit()`` and ``Transformer.transform()``
+-------------------------------------------------------------
+
+:func:`sagemaker.tuner.HyperparameterTuner.fit` and :func:`sagemaker.transformer.Transformer.transform` now wait
+until the completion of the Hyperparameter Tuning Job or Batch Transform Job, respectively.
+To make the function non-blocking, use ``wait=False``.
+
+XGBoost Predictor
+-----------------
+
+The default serializer of ``sagemaker.xgboost.model.XGBoostPredictor`` has been changed from ``NumpySerializer`` to ``LibSVMSerializer``.
+
+
 Parameter and Class Name Changes
 ================================
 
@@ -218,6 +269,8 @@ The follow serializer/deserializer classes have been renamed and/or moved:
 | ``sagemaker.predictor._JsonDeserializer``              | ``sagemaker.deserializers.JSONDeserializer``          |
 +--------------------------------------------------------+-------------------------------------------------------+
 
+``sagemaker.serializers.LibSVMSerializer`` has been added in v2.0.
+
 ``distributions``
 ~~~~~~~~~~~~~~~~~
 
@@ -281,6 +334,19 @@ TensorFlow Serving Predictor
 (For the previous implementation of that class, see `Deprecate Legacy TensorFlow <#deprecate-legacy-tensorflow>`_).
 
 
+Inputs
+------
+
+``s3_input``
+~~~~~~~~~~~~
+
+``sagemaker.session.s3_input`` has been renamed to :class:`sagemaker.inputs.TrainingInput`.
+
+``ShuffleConfig``
+~~~~~~~~~~~~~~~~~
+
+``sagemaker.session.ShuffleConfig`` has been renamed to :class:`sagemaker.inputs.ShuffleConfig`.
+
 Airflow
 -------
 

src/sagemaker/automl/automl.py

@@ -337,6 +337,8 @@ def deploy(
         self,
         initial_instance_count,
         instance_type,
+        serializer=None,
+        deserializer=None,
         candidate=None,
         sagemaker_session=None,
         name=None,
@@ -356,6 +358,16 @@ def deploy(
                 in the ``Endpoint`` created from this ``Model``.
             instance_type (str): The EC2 instance type to deploy this Model to.
                 For example, 'ml.p2.xlarge'.
+            serializer (:class:`~sagemaker.serializers.BaseSerializer`): A
+                serializer object, used to encode data for an inference endpoint
+                (default: None). If ``serializer`` is not None, then
+                ``serializer`` will override the default serializer. The
+                default serializer is set by the ``predictor_cls``.
+            deserializer (:class:`~sagemaker.deserializers.BaseDeserializer`): A
+                deserializer object, used to decode data from an inference
+                endpoint (default: None). If ``deserializer`` is not None, then
+                ``deserializer`` will override the default deserializer. The
+                default deserializer is set by the ``predictor_cls``.
             candidate (CandidateEstimator or dict): a CandidateEstimator used for deploying
                 to a SageMaker Inference Pipeline. If None, the best candidate will
                 be used. If the candidate input is a dict, a CandidateEstimator will be
@@ -405,6 +417,8 @@ def deploy(
         return model.deploy(
             initial_instance_count=initial_instance_count,
             instance_type=instance_type,
+            serializer=serializer,
+            deserializer=deserializer,
             endpoint_name=endpoint_name,
             tags=tags,
             wait=wait,

src/sagemaker/cli/__init__.py

@@ -0,0 +1,14 @@
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying athis file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+"""Tools for automating code updates"""
+from __future__ import absolute_import

src/sagemaker/cli/compatibility/v2/ast_transformer.py

@@ -35,6 +35,7 @@
     modifiers.renamed_params.SessionCreateEndpointImageURIRenamer(),
     modifiers.training_params.TrainPrefixRemover(),
     modifiers.training_input.TrainingInputConstructorRefactor(),
+    modifiers.training_input.ShuffleConfigModuleRenamer(),
     modifiers.serde.SerdeConstructorRenamer(),
 ]
 
@@ -51,6 +52,7 @@
     modifiers.predictors.PredictorImportFromRenamer(),
     modifiers.tfs.TensorFlowServingImportFromRenamer(),
     modifiers.training_input.TrainingInputImportFromRenamer(),
+    modifiers.training_input.ShuffleConfigImportFromRenamer(),
     modifiers.serde.SerdeImportFromAmazonCommonRenamer(),
     modifiers.serde.SerdeImportFromPredictorRenamer(),
 ]

src/sagemaker/cli/compatibility/v2/modifiers/training_input.py

@@ -100,3 +100,73 @@ def modify_node(self, node):
             if node.module == "sagemaker.session":
                 node.module = "sagemaker.inputs"
         return node
+
+
+class ShuffleConfigModuleRenamer(Modifier):
+    """A class to change ``ShuffleConfig`` usage to use ``sagemaker.inputs.ShuffleConfig``."""
+
+    def node_should_be_modified(self, node):
+        """Checks if the ``ast.Call`` node instantiates a class of interest.
+
+        This looks for the following calls:
+
+        - ``sagemaker.session.ShuffleConfig``
+        - ``session.ShuffleConfig``
+
+        Args:
+            node (ast.Call): a node that represents a function call. For more,
+                see https://docs.python.org/3/library/ast.html#abstract-grammar.
+
+        Returns:
+            bool: If the ``ast.Call`` instantiates a class of interest.
+        """
+        if isinstance(node.func, ast.Name):
+            return False
+
+        return matching.matches_name_or_namespaces(
+            node, "ShuffleConfig", ("sagemaker.session", "session")
+        )
+
+    def modify_node(self, node):
+        """Modifies the ``ast.Call`` node to call ``sagemaker.inputs.ShuffleConfig``.
+
+        Args:
+            node (ast.Call): a node that represents a ``sagemaker.session.ShuffleConfig``
+                constructor.
+
+        Returns:
+            ast.Call: the original node, with its namespace changed to use the ``inputs`` module.
+        """
+        _rename_namespace(node, "session")
+        return node
+
+
+class ShuffleConfigImportFromRenamer(Modifier):
+    """A class to update import statements of ``ShuffleConfig``."""
+
+    def node_should_be_modified(self, node):
+        """Checks if the import statement imports ``sagemaker.session.ShuffleConfig``.
+
+        Args:
+            node (ast.ImportFrom): a node that represents a ``from ... import ... `` statement.
+                For more, see https://docs.python.org/3/library/ast.html#abstract-grammar.
+
+        Returns:
+            bool: If the import statement imports ``sagemaker.session.ShuffleConfig``.
+        """
+        return node.module == "sagemaker.session" and any(
+            name.name == "ShuffleConfig" for name in node.names
+        )
+
+    def modify_node(self, node):
+        """Changes the ``ast.ImportFrom`` node's namespace to ``sagemaker.inputs``.
+
+        Args:
+            node (ast.ImportFrom): a node that represents a ``from ... import ... `` statement.
+                For more, see https://docs.python.org/3/library/ast.html#abstract-grammar.
+
+        Returns:
+            ast.ImportFrom: the original node, with its module modified to ``"sagemaker.inputs"``.
+        """
+        node.module = "sagemaker.inputs"
+        return node