g3doc/tutorials/google_cloud_project_setup_instructions.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3t3TAdQCmX9o"
      },
      "source": [
        "##### Copyright 2021 The TensorFlow Cloud Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "lmzbHQeXmYif"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ehIjNHm-E-7"
      },
      "source": [
        "# Setting Up and Connecting To Your Google Cloud Account\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/cloud/tutorials/google_cloud_project_setup_instructions.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/cloud/blob/master/g3doc/tutorials/google_cloud_project_setup_instructions.ipynb\"\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/cloud/blob/master/g3doc/tutorials/google_cloud_project_setup_instructions.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/cloud/tutorials/google_cloud_project_setup_instructions.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "        \u003ca href=\"https://kaggle.com/kernels/welcome?src=https://github.com/tensorflow/cloud/blob/master/g3doc/tutorials/google_cloud_project_setup_instructions.ipynb\" target=\"blank\"\u003e\n",
        "            \u003cimg width=\"90\" src=\"https://www.kaggle.com/static/images/site-logo.png\" alt=\"Kaggle logo\"\u003eRun in Kaggle\n",
        "        \u003c/a\u003e\n",
        "     \u003c/td\u003e\n",
        "\u003c/table\u003e\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXUkgQhef4Op"
      },
      "source": [
        "This guide is to help first time users set up a Google Cloud Platform account specifically with the intention to use [tensorflow_cloud](https://tensorflow.org/cloud) to easily run training at scale on Google Cloud AI Platform. [TensorFlow Cloud](https://tensorflow.org/cloud) provides APIs that allow users to easily go from debugging, training, tuning Keras and TensorFlow code in a local or kaggle environment to distributed training/tuning on Cloud."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j71W0AgV-E_B"
      },
      "source": [
        "## 1. Sign up for Google Cloud Platform\n",
        "\n",
        "To start go to https://cloud.google.com/ and click on “Get Started For Free\". This is a two step sign up process where you will need to provide your name, address and a credit card. The starter account is free and it comes with $300 credit that you can use. For this step you will need to provide a Google Account ( i.e. your Gmail account) to sign in.\n",
        "\n",
        "After completing the sign up process you will be redirected to [Google Cloud Platform welcome page](https://console.cloud.google.com/home/dashboard). click on the \"Home\" tab and make a note of your Project ID and Project number. (see [Identifying projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "WPew3GoCcdtm"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " \n"
          ]
        }
      ],
      "source": [
        "GCP_PROJECT_ID = 'YOUR_PROJECT_ID'\n",
        "PROJECT_NUMBER = 'YOUR_PROJECT_NUMBER'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qUySXAGQ-E_C"
      },
      "source": [
        "## 2.Enable Google Cloud SDK and Authenticate your notebook\n",
        "Now that you have set up your project we can continue with the rest of the configuration steps directly from this notebook. There Three types of notebook, this step (step #2) is slightly different for each notebook, the rest of the steps (step #3 to #6) are the same for all notebooks.\n",
        "\n",
        "* 2.1. Auth for Kaggle notebooks\n",
        "* 2.2. Auth for Colab notebook\n",
        "* 2.3. Auth for Cloud AI Notebooks - Not supported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWfddtzJ-E_C"
      },
      "source": [
        "### 2.1 Setup Auth for a Kaggle Notebook\n",
        "\n",
        "If you are using a Kaggle notebook you will need to repeat this step for any new notebook that you use with this project. In your notebook click on Add-ons -\u003e Google Cloud SDK and follow the instructions on the prompt. Then run the following command to add your auth credentials to this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wv-TIuo_-E_D"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "if \"kaggle_secrets\" in sys.modules:\n",
        "    from kaggle_secrets import UserSecretsClient\n",
        "    UserSecretsClient().set_gcloud_credentials(project=GCP_PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k56ZcFMd-E_D"
      },
      "source": [
        "### 2.2 Setup Auth for a Colab Notebook\n",
        "If you are using a Colab notebook you will need to repeat this step for any new notebook that you use with this project. Run the following command to add your auth credentials to this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "09l5I-Pw-E_E"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ft44j-aA-E_E"
      },
      "source": [
        "## 3. Link your billing account to your project\n",
        "Next step is to [set up  the billing account](https://cloud.google.com/billing/docs/how-to/modify-project) for this project. Google Cloud Creates a project for you by default which is called “My First Project”. We will use this default project. Use your Project ID (from step 1) to run the following commands. This will show you your Billing Account_ID, make a note of this for the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T8FCFUaQ-E_E"
      },
      "outputs": [],
      "source": [
        "!gcloud beta billing accounts list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMqNVFx6-E_F"
      },
      "source": [
        "Use your Billing Account_ID from above and run the following to link your billing account with your project. \n",
        "\n",
        "Note if you use an existing project you may not see an Account_ID, this means you do not have the proper permissions to run the following commands, contact your admin or create a new project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7xQsgsci-E_F"
      },
      "outputs": [],
      "source": [
        "BILLING_ACCOUNT_ID = 'YOUR_BILLING_ACCOUNT_ID'\n",
        "\n",
        "!gcloud beta billing projects link $GCP_PROJECT_ID --billing-account $BILLING_ACCOUNT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wHYb7iA-E_F"
      },
      "source": [
        "## 4. Enable Required APIs for tensorflow-cloud in your project\n",
        "For tensorflow_cloud we use two specific APIs: [AI Platform Training Jobs API](https://console.cloud.google.com/marketplace/product/google/ml.googleapis.com) and [Cloud builder API]( https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview). Note that this is a one time setup for this project, you do not need to rerun this command for every notebook.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X6GFmIJK-E_G"
      },
      "outputs": [],
      "source": [
        "!gcloud services --project $GCP_PROJECT_ID enable ml.googleapis.com cloudbuild.googleapis.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3HqQlmn-E_G"
      },
      "source": [
        "## 5. Create a Google Cloud Storage bucket\n",
        "\n",
        "We will use this storage bucket for temporary assets as well as to save the model checkpoints. Make a note of the name of the bucket for future reference. Note bucket names are unique globally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cQ4kxTtP-E_G"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = 'YOUR_BUCKET_NAME'\n",
        "\n",
        "GCS_BUCKET = f'gs://{BUCKET_NAME}'\n",
        "!gsutil mb -p $GCP_PROJECT_ID $GCS_BUCKET"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jZG5cCJ-E_G"
      },
      "source": [
        "## Create a service account for HP Tuning jobs\n",
        "This step is required to use HP Tuning on Google Cloud using CloudTuner.\n",
        "To [create a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-gcloud) and give it project editor access run the following command and make a note of your service account name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FBX2IT3x-E_G"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT_NAME ='YOUR_SERVICE_ACCOUNT_NAME'\n",
        "\n",
        "SERVICE_ACCOUNT_EMAIL = f'{SERVICE_ACCOUNT_NAME}@{GCP_PROJECT_ID}.iam.gserviceaccount.com'\n",
        "!gcloud iam --project $GCP_PROJECT_ID service-accounts create $SERVICE_ACCOUNT_NAME\n",
        "!gcloud projects add-iam-policy-binding $GCP_PROJECT_ID \\\n",
        "    --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \\\n",
        "    --role=roles/editor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-fNtK6rvGmg"
      },
      "source": [
        "The [`default AI Platform service account`](https://cloud.google.com/ai-platform/training/docs/custom-service-account#default) is identified by an email address with the format `service-PROJECT_NUMBER@cloud-ml.google.com.iam.gserviceaccount.com`. Using your Project number from step one, we construct the service account email and grant the [`default AI Platform service account`](https://cloud.google.com/ai-platform/training/docs/custom-service-account#default) admin role (roles/iam.serviceAccountAdmin) on your new service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AYPnCTq_z7p4"
      },
      "outputs": [],
      "source": [
        "DEFAULT_AI_PLATFORM_SERVICE_ACCOUNT = f'service-{PROJECT_NUMBER}@cloud-ml.google.com.iam.gserviceaccount.com'\n",
        "\n",
        "!gcloud iam --project $GCP_PROJECT_ID service-accounts add-iam-policy-binding \\\n",
        "--role=roles/iam.serviceAccountAdmin \\\n",
        "--member=serviceAccount:$DEFAULT_AI_PLATFORM_SERVICE_ACCOUNT \\\n",
        "$SERVICE_ACCOUNT_EMAIL"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqbjpPt_-E_H"
      },
      "source": [
        "## Congratulations !\n",
        "You are now ready to run tensorflow-cloud. Note that these steps only need to be run one time. Once you have your project setup you can reuse the same project and bucket configuration for future runs. For any new notebooks you will need to repeat the step two to add your Google Cloud auth credentials.\n",
        "\n",
        "Make a note of the following values as they are needed to run tensorflow-cloud."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yE1gpL8oUdV8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your GCP_PROJECT_ID is:       YOUR_PROJECT_ID\n",
            "Your SERVICE_ACCOUNT_NAME is: YOUR_SERVICE_ACCOUNT_NAME\n",
            "Your BUCKET_NAME is:          YOUR_BUCKET_NAME\n"
          ]
        }
      ],
      "source": [
        "print(f\"Your GCP_PROJECT_ID is:       {GCP_PROJECT_ID}\")\n",
        "print(f\"Your SERVICE_ACCOUNT_NAME is: {SERVICE_ACCOUNT_NAME}\")\n",
        "print(f\"Your BUCKET_NAME is:          {BUCKET_NAME}\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "google_cloud_project_setup_instructions.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}