External_data_Local_Files,_Drive,_Sheets,_and_Cloud_Storage.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "External data: Local Files, Drive, Sheets, and Cloud Storage",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/BritneyMuller/colab-notebooks/blob/master/External_data_Local_Files%2C_Drive%2C_Sheets%2C_and_Cloud_Storage.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7Z2jcRKwUHqV"
      },
      "source": [
        "This notebook provides recipes for loading and saving data from external sources."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eikfzi8ZT_rW"
      },
      "source": [
        "# Local file system"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BaCkyg5CV5jF"
      },
      "source": [
        "## Uploading files from your local file system\n",
        "\n",
        "`files.upload` returns a dictionary of the files which were uploaded.\n",
        "The dictionary is keyed by the file name and values are the data which were uploaded."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "vz-jH8T_Uk2c",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "\n",
        "for fn in uploaded.keys():\n",
        "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
        "      name=fn, length=len(uploaded[fn])))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hauvGV4hV-Mh"
      },
      "source": [
        "## Downloading files to your local file system\n",
        "\n",
        "`files.download` will invoke a browser download of the file to your local computer.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "p2E4EKhCWEC5",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "with open('example.txt', 'w') as f:\n",
        "  f.write('some content')\n",
        "\n",
        "files.download('example.txt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "c2W5A2px3doP"
      },
      "source": [
        "# Google Drive\n",
        "\n",
        "You can access files in Drive in a number of ways, including:\n",
        "- Mounting your Google Drive in the runtime's virtual machine\n",
        "- Using a wrapper around the API such as [PyDrive](https://gsuitedevs.github.io/PyDrive/docs/build/html/index.html)\n",
        "- Using the [native REST API](https://developers.google.com/drive/v3/web/about-sdk)\n",
        "\n",
        "\n",
        "\n",
        "Examples of each are below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "u22w3BFiOveA"
      },
      "source": [
        "## Mounting Google Drive locally\n",
        "\n",
        "The example below shows how to mount your Google Drive on your runtime using an authorization code, and how to write and read files there. Once executed, you will be able to see the new file (`foo.txt`) at [https://drive.google.com/](https://drive.google.com/).\n",
        "\n",
        "This only supports reading, writing, and moving files; to programmatically modify sharing settings or other metadata, use one of the other options below.\n",
        "\n",
        "**Note:** When using the 'Mount Drive' button in the file browser, no authentication codes are necessary for notebooks that have only been edited by the current user."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RWSJpsyKqHjH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 109
        },
        "outputId": "15bd239f-5a6a-447d-dd22-01ee5398b991"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XDg9OBaYqRMd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "5df8b65d-20d2-4956-b8a7-e99d4579d567"
      },
      "source": [
        "with open('/content/drive/My Drive/foo.txt', 'w') as f:\n",
        "  f.write('Hello Google Drive!')\n",
        "!cat /content/drive/My\\ Drive/foo.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Hello Google Drive!"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "D78AM1fFt2ty",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f7117bc5-2ffd-4c0a-8809-f9fed79b4682"
      },
      "source": [
        "drive.flush_and_unmount()\n",
        "print('All changes made in this colab session should now be visible in Drive.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "All changes made in this colab session should now be visible in Drive.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7taylj9wpsA2"
      },
      "source": [
        "## PyDrive\n",
        "\n",
        "The examples below demonstrate authentication and file upload/download using PyDrive. More examples are available in the [PyDrive documentation](https://gsuitedevs.github.io/PyDrive/docs/build/html/index.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zU5b6dlRwUQk",
        "colab": {}
      },
      "source": [
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XkHZtRhm09QS"
      },
      "source": [
        "Authenticate and create the PyDrive client.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "1w7lrGcW08Ds",
        "colab": {}
      },
      "source": [
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZF3Topk31DCF"
      },
      "source": [
        "Create and upload a text file.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "vBuUW-q41tb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e5558ef2-e5db-4e4e-9f01-186e9dd4bc26"
      },
      "source": [
        "uploaded = drive.CreateFile({'title': 'Sample upload.txt'})\n",
        "uploaded.SetContentString('Sample upload file content')\n",
        "uploaded.Upload()\n",
        "print('Uploaded file with ID {}'.format(uploaded.get('id')))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Uploaded file with ID 14vDAdqp7BSCQnoougmgylBexIr2AQx2T\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bbOFN3d_1tcG"
      },
      "source": [
        "Load a file by ID and print its contents.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eFOvsAYk1tcH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c8f0e09a-f873-4c43-8211-5f0a0c5e0e5f"
      },
      "source": [
        "downloaded = drive.CreateFile({'id': uploaded.get('id')})\n",
        "print('Downloaded content \"{}\"'.format(downloaded.GetContentString()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloaded content \"Sample upload file content\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jRQ5_yMcqJiV"
      },
      "source": [
        "## Drive REST API\n",
        "\n",
        "In order to use the Drive API, we must first authenticate and construct an API client.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "r-exJtdG3XwJ",
        "colab": {}
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "from googleapiclient.discovery import build\n",
        "drive_service = build('drive', 'v3')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eDLm7MHQEr2U"
      },
      "source": [
        "With this client, we can use any of the functions in the [Google Drive API reference](https://developers.google.com/drive/v3/reference/). Examples follow.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bRFyEsdfBxJ9"
      },
      "source": [
        "### Creating a new Drive file with data from Python\n",
        "\n",
        "First, create a local file to upload."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "F1-nafvN-NwW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "4466d27b-edd0-4ef1-8a10-0e180a231af2"
      },
      "source": [
        "with open('/tmp/to_upload.txt', 'w') as f:\n",
        "  f.write('my sample file')\n",
        "\n",
        "print('/tmp/to_upload.txt contains:')\n",
        "!cat /tmp/to_upload.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/tmp/to_upload.txt contains:\n",
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "r5X77CZ5_G-C"
      },
      "source": [
        "Upload it using the [`files.create`](https://developers.google.com/drive/v3/reference/files/create) method. Further details on uploading files are available in the [developer documentation](https://developers.google.com/drive/v3/web/manage-uploads)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3Jv6jh6HEpP8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "cfc1186f-68a1-4620-a18f-99bc6d83e82e"
      },
      "source": [
        "from googleapiclient.http import MediaFileUpload\n",
        "\n",
        "file_metadata = {\n",
        "  'name': 'Sample file',\n",
        "  'mimeType': 'text/plain'\n",
        "}\n",
        "media = MediaFileUpload('/tmp/to_upload.txt', \n",
        "                        mimetype='text/plain',\n",
        "                        resumable=True)\n",
        "created = drive_service.files().create(body=file_metadata,\n",
        "                                       media_body=media,\n",
        "                                       fields='id').execute()\n",
        "print('File ID: {}'.format(created.get('id')))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ID: 1Cw9CqiyU6zbXFD9ViPZu_3yX-sYF4W17\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "j5VyISCKFrqU"
      },
      "source": [
        "After executing the cell above, you will see a new file named 'Sample file' at [https://drive.google.com/](https://drive.google.com/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "P3KX0Sm0E2sF"
      },
      "source": [
        "### Downloading data from a Drive file into Python\n",
        "\n",
        "Download the file we uploaded above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hEzpzWYWsYKg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "57c1ac51-a8ec-497e-db77-f34fd974ac2f"
      },
      "source": [
        "file_id = created.get('id')\n",
        "\n",
        "import io\n",
        "from googleapiclient.http import MediaIoBaseDownload\n",
        "\n",
        "request = drive_service.files().get_media(fileId=file_id)\n",
        "downloaded = io.BytesIO()\n",
        "downloader = MediaIoBaseDownload(downloaded, request)\n",
        "done = False\n",
        "while done is False:\n",
        "  # _ is a placeholder for a progress object that we ignore.\n",
        "  # (Our file is small, so we skip reporting progress.)\n",
        "  _, done = downloader.next_chunk()\n",
        "\n",
        "downloaded.seek(0)\n",
        "print('Downloaded file contents are: {}'.format(downloaded.read()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloaded file contents are: b'my sample file'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ImztleG2Ciae"
      },
      "source": [
        "In order to download a different file, set `file_id` above to the ID of that file, which will look like \"1uBtlaggVyWshwcyP6kEI-y_W3P8D26sz\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sOm9PFrT8mGG"
      },
      "source": [
        "# Google Sheets\n",
        "\n",
        "Our examples below use the open-source [`gspread`](https://github.com/burnash/gspread) library for interacting with Google Sheets.\n",
        "\n",
        "First, install the package using `pip`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Mwu_sWHv4jEo",
        "colab": {}
      },
      "source": [
        "!pip install --upgrade --quiet gspread"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qzi9VsEqzI-o"
      },
      "source": [
        "Import the library, authenticate, and create the interface to Sheets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6d0xJz3VzLOo",
        "colab": {}
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "\n",
        "import gspread\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "gc = gspread.authorize(GoogleCredentials.get_application_default())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yjrZQUrt6kKj"
      },
      "source": [
        "Below is a small set of `gspread` examples. Additional examples are availabe at the [`gspread` GitHub page](https://github.com/burnash/gspread#more-examples)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WgXqE02UofZG"
      },
      "source": [
        "## Creating a new sheet with data from Python"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tnnYKhGfzGeP",
        "colab": {}
      },
      "source": [
        "sh = gc.create('My cool spreadsheet')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v9Ia9JVc6Zvk"
      },
      "source": [
        "After executing the cell above, you will see a new spreadsheet named 'My cool spreadsheet' at [https://sheets.google.com](https://sheets.google.com/).\n",
        "\n",
        "Open our new sheet and add some random data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ASdgLIt2s-ux",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "68c9eed9-bab9-4169-c5b7-769f7b9e0194"
      },
      "source": [
        "worksheet = gc.open('My cool spreadsheet').sheet1\n",
        "\n",
        "cell_list = worksheet.range('A1:C2')\n",
        "\n",
        "import random\n",
        "for cell in cell_list:\n",
        "  cell.value = random.randint(1, 10)\n",
        "\n",
        "worksheet.update_cells(cell_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'spreadsheetId': '1dsQeN0YzXuM387l_CuyEbsYzL2ew9TJFzR-E-RQnwxs',\n",
              " 'updatedCells': 6,\n",
              " 'updatedColumns': 3,\n",
              " 'updatedRange': 'Sheet1!A1:C2',\n",
              " 'updatedRows': 2}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 0
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k9q0pp33dckN"
      },
      "source": [
        "## Downloading data from a sheet into Python as a Pandas DataFrame\n",
        "\n",
        "Read back the random data that we inserted above and convert the result into a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "J4QxBareshEV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "outputId": "baf17cd6-3c3d-4a34-b1f9-642516cb2bfe"
      },
      "source": [
        "worksheet = gc.open('My cool spreadsheet').sheet1\n",
        "\n",
        "# get_all_values gives a list of rows.\n",
        "rows = worksheet.get_all_values()\n",
        "print(rows)\n",
        "\n",
        "import pandas as pd\n",
        "pd.DataFrame.from_records(rows)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['6', '3', '4'], ['7', '2', '1']]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.module+javascript": "\n      import \"https://ssl.gstatic.com/colaboratory/data_table/81868506e94e6988/data_table.js\";\n\n      window.createDataTable({\n        data: [[{\n            'v': 0,\n            'f': \"0\",\n        },\n\"6\",\n\"3\",\n\"4\"],\n [{\n            'v': 1,\n            'f': \"1\",\n        },\n\"7\",\n\"2\",\n\"1\"]],\n        columns: [[\"number\", \"index\"], [\"string\", \"0\"], [\"string\", \"1\"], [\"string\", \"2\"]],\n        rowsPerPage: 25,\n      });\n    ",
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   0  1  2\n",
              "0  6  3  4\n",
              "1  7  2  1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 0
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "S7c8WYyQdh5i"
      },
      "source": [
        "# Google Cloud Storage (GCS)\n",
        "\n",
        "In order to use Colaboratory with GCS, you'll need to create a [Google Cloud project](https://cloud.google.com/storage/docs/projects) or use a pre-existing one.\n",
        "\n",
        "Specify your project ID below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VYC5CyAbAtU7",
        "colab": {}
      },
      "source": [
        "project_id = 'Your_project_ID_here'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5iRYBSVCbAlK"
      },
      "source": [
        "Files in GCS are contained in [buckets](https://cloud.google.com/storage/docs/key-terms#buckets).\n",
        "\n",
        "Buckets must have a globally-unique name, so we generate one here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JgjL1e0ua-kN",
        "colab": {}
      },
      "source": [
        "import uuid\n",
        "bucket_name = 'colab-sample-bucket-' + str(uuid.uuid1())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-OtHMRVda9pJ"
      },
      "source": [
        "In order to access GCS, we must authenticate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NQX0hbfYaEKc",
        "colab": {}
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YLPAbDDFaLXU"
      },
      "source": [
        "GCS can be accessed via the `gsutil` command-line utility or via the native Python API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TvuZEyRQXmyS"
      },
      "source": [
        "## `gsutil`\n",
        "\n",
        "First, we configure `gsutil` to use the project we specified above by using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TpnuFITI6Tzu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "04f1dd6d-4d7e-4264-b37b-1e8f645d6d38"
      },
      "source": [
        "!gcloud config set project {project_id}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Updated property [core/project].\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NAM6vyXAfVUj"
      },
      "source": [
        "Create a local file to upload."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LADpx7LReOMk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "46db7cfa-9ad8-405b-f715-b466a3b2cb7a"
      },
      "source": [
        "with open('/tmp/to_upload.txt', 'w') as f:\n",
        "  f.write('my sample file')\n",
        "\n",
        "print('/tmp/to_upload.txt contains:')\n",
        "!cat /tmp/to_upload.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/tmp/to_upload.txt contains:\n",
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sSz0aInzWXFI"
      },
      "source": [
        "Make a bucket to which we'll upload the file ([documentation](https://cloud.google.com/storage/docs/gsutil/commands/mb))."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Bcpvh_R_6jKB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e1c132b8-6a5c-46db-b1fa-f5768089890c"
      },
      "source": [
        "!gsutil mb gs://{bucket_name}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Creating gs://colab-sample-bucket-44971372-baaf-11e7-ae30-0242ac110002/...\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MHPM16gWWNvn"
      },
      "source": [
        "Copy the file to our new bucket ([documentation](https://cloud.google.com/storage/docs/gsutil/commands/cp))."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "L5cMl7XV65be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 92
        },
        "outputId": "bb51e51d-7f5f-4e2b-935c-203b8d314115"
      },
      "source": [
        "!gsutil cp /tmp/to_upload.txt gs://{bucket_name}/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Copying file:///tmp/to_upload.txt [Content-Type=text/plain]...\n",
            "/ [1 files][   14.0 B/   14.0 B]                                                \n",
            "Operation completed over 1 objects/14.0 B.                                       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xAPlMk6nWKNm"
      },
      "source": [
        "Dump the contents of our newly copied file to make sure everything worked ([documentation](https://cloud.google.com/storage/docs/gsutil/commands/cat)).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pJGU6gX-7M-N",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "38db2bd3-0879-4a2e-8f41-c9495ba570a9"
      },
      "source": [
        "!gsutil cat gs://{bucket_name}/to_upload.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "qoectKiGOABf",
        "colab": {
          "height": 34
        },
        "outputId": "97b483ac-c2b1-46f0-f4ac-61b3701a8c89"
      },
      "source": [
        "#@markdown Once the upload has finished, the data will appear in the Cloud Console storage browser for your project:\n",
        "print('https://console.cloud.google.com/storage/browser?project=' + project_id)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "https://console.cloud.google.com/storage/browser?project=Your_project_ID_here\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Q2CWQGIghDux"
      },
      "source": [
        "Finally, we'll download the file we just uploaded in the example above. It's as simple as reversing the order in the `gsutil cp` command."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "lPdTf-6O73ll",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "outputId": "a6da299e-00ff-42a7-f845-9a93a4ccce61"
      },
      "source": [
        "!gsutil cp gs://{bucket_name}/to_upload.txt /tmp/gsutil_download.txt\n",
        "  \n",
        "# Print the result to make sure the transfer worked.\n",
        "!cat /tmp/gsutil_download.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Copying gs://colab-sample-bucket483f20dc-baaf-11e7-ae30-0242ac110002/to_upload.txt...\n",
            "/ [1 files][   14.0 B/   14.0 B]                                                \n",
            "Operation completed over 1 objects/14.0 B.                                       \n",
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0ENMqxq25szn"
      },
      "source": [
        "## Python API\n",
        "\n",
        "These snippets based on [a larger example](https://github.com/GoogleCloudPlatform/storage-file-transfer-json-python/blob/master/chunked_transfer.py) that shows additional uses of the API.\n",
        "\n",
        " First, we create the service client."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wkJ5ShIgceqI",
        "colab": {}
      },
      "source": [
        "from googleapiclient.discovery import build\n",
        "gcs_service = build('storage', 'v1')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "R1Hyoh07gQHY"
      },
      "source": [
        "Create a local file to upload."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "s1htTNVlgQHe",
        "colab": {
          "height": 51
        },
        "outputId": "1a002b58-6c4d-46d0-9572-a43b9bccbfe8"
      },
      "source": [
        "with open('/tmp/to_upload.txt', 'w') as f:\n",
        "  f.write('my sample file')\n",
        "\n",
        "print('/tmp/to_upload.txt contains:')\n",
        "!cat /tmp/to_upload.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/tmp/to_upload.txt contains:\n",
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YJKBHaUlgDhl"
      },
      "source": [
        "Create a bucket in the project specified above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YFVbF4cdhd9Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ffa2a4ec-ee02-4fc2-8ed4-a4d73d04e6be"
      },
      "source": [
        "# Use a different globally-unique bucket name from the gsutil example above.\n",
        "import uuid\n",
        "bucket_name = 'colab-sample-bucket-' + str(uuid.uuid1())\n",
        "\n",
        "body = {\n",
        "  'name': bucket_name,\n",
        "  # For a full list of locations, see:\n",
        "  # https://cloud.google.com/storage/docs/bucket-locations\n",
        "  'location': 'us',\n",
        "}\n",
        "gcs_service.buckets().insert(project=project_id, body=body).execute()\n",
        "print('Done')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ppkrR7p4mx_P"
      },
      "source": [
        "Upload the file to our newly created bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cFAq-F2af5TJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d07f7059-5767-4d58-ac71-e457a76e8c07"
      },
      "source": [
        "from googleapiclient.http import MediaFileUpload\n",
        "\n",
        "media = MediaFileUpload('/tmp/to_upload.txt', \n",
        "                        mimetype='text/plain',\n",
        "                        resumable=True)\n",
        "\n",
        "request = gcs_service.objects().insert(bucket=bucket_name, \n",
        "                                       name='to_upload.txt',\n",
        "                                       media_body=media)\n",
        "\n",
        "response = None\n",
        "while response is None:\n",
        "  # _ is a placeholder for a progress object that we ignore.\n",
        "  # (Our file is small, so we skip reporting progress.)\n",
        "  _, response = request.next_chunk()\n",
        "\n",
        "print('Upload complete')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Upload complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "Lvz1BspfpvTl",
        "colab": {
          "height": 34
        },
        "outputId": "97b483ac-c2b1-46f0-f4ac-61b3701a8c89"
      },
      "source": [
        "#@markdown Once the upload has finished, the data will appear in the Cloud Console storage browser for your project:\n",
        "print('https://console.cloud.google.com/storage/browser?project=' + project_id)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "https://console.cloud.google.com/storage/browser?project=Your_project_ID_here\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "P6aWjfTv7bit"
      },
      "source": [
        "Download the file we just uploaded."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "z1_FuDjAozF1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ab14cf25-7b51-41c9-d88c-9a94f5c79dfc"
      },
      "source": [
        "from apiclient.http import MediaIoBaseDownload\n",
        "\n",
        "with open('/tmp/downloaded_from_gcs.txt', 'wb') as f:\n",
        "  request = gcs_service.objects().get_media(bucket=bucket_name,\n",
        "                                            object='to_upload.txt')\n",
        "  media = MediaIoBaseDownload(f, request)\n",
        "\n",
        "  done = False\n",
        "  while not done:\n",
        "    # _ is a placeholder for a progress object that we ignore.\n",
        "    # (Our file is small, so we skip reporting progress.)\n",
        "    _, done = media.next_chunk()\n",
        "\n",
        "print('Download complete')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Download complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OnjWcDFogdY2"
      },
      "source": [
        "Inspect the downloaded file.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DxLyhaiBpAGX",
        "colab": {
          "height": 34
        },
        "outputId": "eb379e18-a6bf-4af2-caf4-f8b9c57f4a80"
      },
      "source": [
        "!cat /tmp/downloaded_from_gcs.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "my sample file"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}