google_earth_engine/COG_Backed_Image_Asset_Registration_into_an_ImageCollection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "COG-Backed Image Asset Registration into an ImageCollection",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "fSIfBsgi8dNK"
      },
      "source": [
        "#@title Copyright 2020 Google LLC. { display-mode: \"form\" }\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrEM35gqHouU"
      },
      "source": [
        "# Cloud GeoTiff Backed Earth Engine Assets\n",
        "\n",
        "***Note:*** *The REST API contains new and advanced features that may not be suitable for all users.  If you are new to Earth Engine, please get started with the [JavaScript guide](https://developers.google.com/earth-engine/getstarted).*\n",
        "\n",
        "Earth Engine can load images from Cloud Optimized GeoTiffs (COGs) in Google Cloud Storage ([learn more](https://developers.google.com/earth-engine/image_overview#images-from-cloud-geotiffs)).  This notebook demonstrates how to create Earth Engine assets backed by COGs.  An advantage of COG-backed assets is that the spatial and metadata fields of the image will be indexed at asset creation time, making the image more performant in collections.  (In contrast, an image created through `ee.Image.loadGeoTIFF` and put into a collection will require a read of the GeoTiff for filtering operations on the collection.)  A disadvantage of COG-backed assets is that they may be several times slower than standard assets when used in computations.\n",
        "\n",
        "To create a COG-backed asset, make a `POST` request to the Earth Engine [`CreateAsset` endpoint](https://developers.google.com/earth-engine/reference/rest/v1alpha/projects.assets/create).  As shown in the following, this request must be authorized to create an asset in your user folder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmxat3ujhwGx"
      },
      "source": [
        "## Start an authorized session\n",
        "\n",
        "To be able to make an Earth Engine asset in your user folder, you need to be able to authenticate as you when you make the request.  The Earth Engine Python authenticator can be leveraged as a client app that is able to pass your credentials along.  Follow the instructions in the cell output to authenticate.  (Note that this auth flow is not supported if this notebook is being run in playgroud mode; make a copy before proceeding).\n",
        "\n",
        "For more details, see [this guide](https://google-auth.readthedocs.io/en/latest/user-guide.html#obtaining-credentials) on obtaining credentials in this manner, [this reference](https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html) on the `Flow` module, [this reference](https://github.com/googleapis/google-api-python-client/blob/master/docs/client-secrets.md) for the client secrets format, and [`oauth.py`](https://github.com/google/earthengine-api/blob/master/python/ee/oauth.py) from the Earth Engine python library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVu8GhINwYfO"
      },
      "source": [
        "# This has details about the Earth Engine Python Authenticator client.\n",
        "from ee import oauth\n",
        "from google_auth_oauthlib.flow import Flow\n",
        "import json\n",
        "\n",
        "# Build the `client_secrets.json` file by borrowing the\n",
        "# Earth Engine python authenticator.\n",
        "client_secrets = {\n",
        "    'web': {\n",
        "        'client_id': oauth.CLIENT_ID,\n",
        "        'client_secret': oauth.CLIENT_SECRET,\n",
        "        'redirect_uris': [oauth.REDIRECT_URI],\n",
        "        'auth_uri': 'https://accounts.google.com/o/oauth2/auth',\n",
        "        'token_uri': 'https://accounts.google.com/o/oauth2/token'\n",
        "    }\n",
        "}\n",
        "\n",
        "# Write to a json file.\n",
        "client_secrets_file = 'client_secrets.json'\n",
        "with open(client_secrets_file, 'w') as f:\n",
        "  json.dump(client_secrets, f, indent=2)\n",
        "\n",
        "# Start the flow using the client_secrets.json file.\n",
        "flow = Flow.from_client_secrets_file(client_secrets_file,\n",
        "    scopes=oauth.SCOPES,\n",
        "    redirect_uri=oauth.REDIRECT_URI)\n",
        "\n",
        "# Get the authorization URL from the flow.\n",
        "auth_url, _ = flow.authorization_url(prompt='consent')\n",
        "\n",
        "# Print instructions to go to the authorization URL.\n",
        "oauth._display_auth_instructions_with_print(auth_url)\n",
        "print('\\n')\n",
        "\n",
        "# The user will get an authorization code.\n",
        "# This code is used to get the access token.\n",
        "code = input('Enter the authorization code: \\n')\n",
        "flow.fetch_token(code=code)\n",
        "\n",
        "# Get an authorized session from the flow.\n",
        "session = flow.authorized_session()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nU2VUnThUtFi"
      },
      "source": [
        "# Authorize Google Cloud SDK so we can use GCS\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GFj3wdnKXXqa"
      },
      "source": [
        "import datetime\n",
        "from pprint import pprint\n",
        "import time\n",
        "import os.path\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u7Te9tQCT-cm"
      },
      "source": [
        "# CHANGE - point to the GCS bucket and path to the source COGs\n",
        "gcs_folder = 'netcdf-cogs-example/sresa1b.ncar_pcm1.1.monthly.tavg.1950-2099'\n",
        "filelist = !gsutil ls gs://{gcs_folder}/*.tiff\n",
        "print(filelist)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t5T_w5RZVOUA"
      },
      "source": [
        "def getDate(filepath):\n",
        "  # This expects the COG files in GCS to be named like *_NNN.tiff, where NNN is number of days since 1/1/1950 \n",
        "  namestring = os.path.basename(filepath)\n",
        "  origin_date = datetime.datetime(1950, 1, 1, tzinfo=datetime.timezone.utc)\n",
        "  match = re.match(r'.*_(\\d+).tiff', namestring)\n",
        "  days = int(match[1])\n",
        "  computed_date = origin_date + datetime.timedelta(days=days)\n",
        "  return computed_date.isoformat()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S826-7H-egko"
      },
      "source": [
        "def getName(filepath):\n",
        "  namestring = os.path.basename(filepath)\n",
        "  asset_name = os.path.splitext(namestring)[0]\n",
        "  \n",
        "  return asset_name.replace('.', '_')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "10rvX51tVtgO"
      },
      "source": [
        "# CHANGE - point to the project folder you want to put the image collection into\n",
        "project_folder = 'example_project_folder'\n",
        "# The ImageCollection path.\n",
        "asset_root = os.path.basename(gcs_folder).replace('.', '_')\n",
        "\n",
        "# Create the ImageCollection\n",
        "request = {\n",
        "  'type': 'IMAGE_COLLECTION',\n",
        "  #'properties': {\n",
        "  #  # it might be useful to include the URL of the original NetCDF, and/or the STAC collection as metadata\n",
        "  #  'source': 'source metadata'\n",
        "  #},\n",
        "  #'startTime': asset_date,\n",
        "  #'endTime': asset_date\n",
        "}\n",
        "url = 'https://earthengine.googleapis.com/v1alpha/projects/{}/assets?assetId={}'\n",
        "response = session.post(\n",
        "  url = url.format(project_folder, asset_root),\n",
        "  data = json.dumps(request),\n",
        "  timeout=120\n",
        ")\n",
        "response.raise_for_status()\n",
        "\n",
        "# Add images to the collection\n",
        "for file in filelist: \n",
        "  print(file)\n",
        "  asset_date = getDate(file)\n",
        "  asset_name = getName(file)\n",
        "\n",
        "  # Prepare the request\n",
        "  request = {\n",
        "    'type': 'IMAGE',\n",
        "    'gcs_location': {\n",
        "      'uris': [file]\n",
        "    },\n",
        "    #'properties': {\n",
        "    #  'source': 'source metadata'\n",
        "    #},\n",
        "    'startTime': asset_date,\n",
        "    'endTime': asset_date\n",
        "  }\n",
        "\n",
        "\n",
        "  # Your collection name and new asset name.\n",
        "  asset_id = asset_root + '/' + asset_name\n",
        "\n",
        "  url = 'https://earthengine.googleapis.com/v1alpha/projects/{}/assets?assetId={}'\n",
        "\n",
        "  response = session.post(\n",
        "    url = url.format(project_folder, asset_id),\n",
        "    data = json.dumps(request),\n",
        "    timeout=120\n",
        "  )\n",
        "\n",
        "  response.raise_for_status()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Details on COG-backed assets\n",
        "\n",
        "### Permissions\n",
        "The ACLs of COG-backed Earth Engine assets and the underlying data are managed separately. If a COG-backed asset is shared in Earth Engine, it is the owner's responsibility to ensure that the data in GCS is shared with the same parties. If the data is not visible, Earth Engine will return an error of the form \"Failed to load the GeoTIFF at `gs://my-bucket/my-object#123456`\" (123456 is the generation of the object).\n",
        "\n",
        "### Generations\n",
        "When a COG-backed asset is created, Earth Engine reads the metadata of the TIFF in Cloud Storage and creates asset store entry. The URI associated with that entry must have a generation.  See the [object versioning docs](https://cloud.google.com/storage/docs/object-versioning) for details on generations. If a generation is specified (e.g., `gs://foo/bar#123`), Earth Engine will use it. If a generation is not specified, Earth Engine will use the latest generation of the object. \n",
        "\n",
        "That means that if the object in GCS is updated, Earth Engine will return a \"Failed to load the GeoTIFF at `gs://my-bucket/my-object#123456`\" error because the expected object no longer exists (unless the bucket enables  multiple object versions).  This policy is designed to keep metadata of the asset in sync with the metadata of the object.  \n",
        "\n",
        "### Configuration\n",
        "In terms of how a COG should be configured, the TIFF MUST be:\n",
        "\n",
        "- Tiled, where the tile dimensions are either:\n",
        "  - 16x16\n",
        "  - 32x32\n",
        "  - 64x64\n",
        "  - 128x128\n",
        "  - 256x256\n",
        "  - 512x512\n",
        "  - 1024x1024\n",
        "\n",
        "- Arranged so that all IFDs are at the beginning.\n",
        "\n",
        "For best performance:\n",
        "\n",
        "- Use tile dimensions of 128x128 or 256x256.\n",
        "- Include power of 2 overviews.\n",
        "\n",
        "See [this page](https://github.com/cogeotiff/rio-cogeo#web-optimized-cog) for more details on an optimized configuration."
      ],
      "metadata": {
        "id": "HAg-vG2ESHTk"
      }
    }
  ]
}