cloud_storage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Big Arrays, Fast: Profiling Cloud Storage Read Throughput\n",
    "\n",
    "By [Ryan Abernathey](https://github.com/rabernat)\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Many geoscience problems involve analyzing hundreds of Gigabytes (GB), many Terabytes (TB = 100 GB), or even a Petabyte (PB = 1000 TB) of data.\n",
    "For such data volumes, downloading data to personal computers is inefficient.\n",
    "Data-proximate computing, possibly using a parallel computing cluster, is one way to overcome this challenge.\n",
    "With data-proximate computing, a user access a remote computer which is connected tightly to shared, high-performance storage, enabling rapid processing without the need for a download step.\n",
    "\n",
    "<img width=\"400px\" src=\"https://ndownloader.figshare.com/files/22017009/preview/22017009/preview.jpg\"/>\n",
    "\n",
    "Figure from _Data Access Modes in Science_, by Ryan Abernathey. http://dx.doi.org/10.6084/M9.FIGSHARE.11987466.V1\n",
    "\n",
    "\n",
    "The cloud offers many exciting new possibilities for scientific data analysis.\n",
    "It also brings new technical challenges.\n",
    "Many scientific users are accustomed to the environments found on High-Performance Computing (HPC) systems, such as NSF's XSEDE Computers or NCAR's Cheyenne system.\n",
    "These systems combine parallel computing with a high-performance shared filesystem, enabling massively parallel data analytics.\n",
    "When shifting workloads to the cloud, one of the biggest challenges is how to store data: _cloud computing does not easily support large shared filesystems, as commonly found on HPC systems_.\n",
    "The preferred way to store data in the cloud is using _cloud object storage_, such as Amazon S3 or Google Cloud Storage.\n",
    "\n",
    "<img width=\"400px\" src=\"https://assets.ubuntu.com/v1/09b510e0-UAS_storage_options.png\">\n",
    "\n",
    "_Image via [Ubuntu Website](https://ubuntu.com/blog/what-are-the-different-types-of-storage-block-object-and-file), ©2020 Canonical Ltd. Ubuntu, Used with Permission_\n",
    "\n",
    "\n",
    "Cloud object storage is essentially a key/value storage system.\n",
    "They keys are strings, and the values are bytes of data.\n",
    "Data is read and written using HTTP calls.\n",
    "The performance of object storage is very different from file storage.\n",
    "On one hand, each individual read / write to object storage has a high overhead (10-100 ms), since it has to go over the network.\n",
    "On the other hand, object storage \"scales out\" nearly infinitely, meaning that we can make hundreds, thousands, or millions of concurrent reads / writes.\n",
    "This makes object storage well suited for distributed data analytics.\n",
    "However, the software architecture of a data analysis system must be adapted to take advantage of these properties.\n",
    "\n",
    "The goal of this notebook is to demonstrate one software stack that can effectively exploit the scalable nature of cloud storage and computing for data processing. There are three key elements:\n",
    "\n",
    "- [Zarr](http://zarr.readthedocs.io/) - a library for the storage of multi-dimensional arrays using compressed chunks.\n",
    "- [Filesystem Spec](https://filesystem-spec.readthedocs.io/) - an abstraction layer for working with different storage technologies. Specific implementations of filesystem spec (e.g. [gcsfs](https://gcsfs.readthedocs.io/), Google Cloud Storage Filesystem) allow Zarr to read and write data from object storage.\n",
    "- [Dask](http://dask.pydata.org/) - a distributed computing framework that allows us to read and write data in parallel asynchronously.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing the Data\n",
    "\n",
    "For this demonstration, we will use data stored in Google Cloud Storage.\n",
    "The data come from the [GFDL CM2.6 high-resolution climate model](https://www.gfdl.noaa.gov/cm2-6/).\n",
    "They are catalogged on the Pangeo Cloud Data Catalog here:\n",
    "\n",
    "- https://catalog.pangeo.io/browse/master/ocean/GFDL_CM2_6/\n",
    "\n",
    "The catalog returns an [http://xarray.pydata.org/](Xarray) object pointing to the data on google cloud:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.array as dsa\n",
    "import fsspec\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from contextlib import contextmanager\n",
    "import pandas as pd\n",
    "import intake\n",
    "import time\n",
    "import seaborn as sns\n",
    "import dask\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intake import open_catalog\n",
    "cat = open_catalog(\"https://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/ocean/GFDL_CM2.6.yaml\")\n",
    "item = cat[\"GFDL_CM2_6_control_ocean\"]\n",
    "ds = item.to_dask()\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example, we are not interested in Xarray's advanced data analytics features, so we will work directly with the raw array data. Here we examine the data from the `temp` variable (ocean temperature):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ds.temp.data\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before dropping down to the dask level, we take a quick peek at the data using Xarray's built-in visualization capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.temp[0, 0].plot(figsize=(16, 8), center=False, robust=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Side note:_ We can open this array directly with dask, bypassing xarray, if we know the path on the object store.\n",
    "(These give identical results.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item.urlpath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsa.from_zarr(fsspec.get_mapper(\"gs://cmip6/GFDL_CM2_6/control/ocean/temp\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some important observations:\n",
    "\n",
    "- The data is organized into 2400 distinct chunks--each one corresponds to an individual object in Google Cloud Storage\n",
    "- The in-memory size of each chunk is 194.4 MB\n",
    "- The chunks are contiguous across the latitude and longitude dimensions (axes 2 and 3) and span 5 vertical levels (axis 1). Each timestep (axis 0) is in a different chunk. \n",
    "- Zarr + Dask know how to virtually combine these chunks into a single 4-dimensional array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set Up Benchmarking\n",
    "\n",
    "Now we want to see how fast we can load data from the object store. _We are not interested in any computations._ Here the goal is simply to measure read performance and how it scales. In order to do this, we  employ a trick: we  store the data into a virtual storage target that simply discards the data, similar to piping data to `/dev/null` on a filesystem.\n",
    "\n",
    "To do this, we define the following storage class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DevNullStore:\n",
    "    \n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def __setitem__(*args, **kwargs):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is basically a dictionary that forgets whatever you put in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "null_store = DevNullStore()\n",
    "# this line produces no error but actually does nothing\n",
    "null_store['foo'] = 'bar'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how long it takes to store one chunk of data into this null storage target.\n",
    "This essentially measures the read time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time dsa.store(data[0, :5], null_store, lock=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to scale this out to see how the read time scales as a function of number of parallel reads.\n",
    "For this, we need a Dask cluster.\n",
    "In our Pangeo environment, we can use [Dask Gateway](https://medium.com/pangeo/pangeo-with-dask-gateway-4b638825f105) to create a Dask cluster.\n",
    "But there is an analogous way to do this on nearly any distributed computing environment.\n",
    "HPC-style environments (PBS, Slurm, etc.) are supported via [Dask Jobqueue](http://jobqueue.dask.org/en/latest/).\n",
    "Most cloud environments can work with [Dask Kubernetes](https://kubernetes.dask.org/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask_gateway import Gateway\n",
    "from dask.distributed import Client\n",
    "\n",
    "gateway = Gateway()\n",
    "cluster = gateway.new_cluster()\n",
    "cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we create a context manager to help us keep track of the results of our benchmarking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DiagnosticTimer:\n",
    "    def __init__(self):\n",
    "        self.diagnostics = []\n",
    "\n",
    "    @contextmanager\n",
    "    def time(self, **kwargs):\n",
    "        tic = time.time()\n",
    "        yield\n",
    "        toc = time.time()\n",
    "        kwargs[\"runtime\"] = toc - tic\n",
    "        self.diagnostics.append(kwargs)\n",
    "\n",
    "    def dataframe(self):\n",
    "        return pd.DataFrame(self.diagnostics)\n",
    "    \n",
    "diag_timer = DiagnosticTimer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do Benchmarking\n",
    "\n",
    "We want to keep track of some information about our array.\n",
    "Here we figure out the size (in bytes) of the chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunksize = np.prod(data.chunksize) * data.dtype.itemsize\n",
    "chunksize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And about how many workers / threads are in the cluster.\n",
    "The current cluster setup uses two threads and approximately 4 GB of memory per worker.\n",
    "These settings are the defaults for the current Pangeo environment and have not been explored extensively.\n",
    "We do not expect high sensitivity to these parameters here, as the task is I/O limited, not CPU limited."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def total_nthreads():\n",
    "    return sum([v for v in client.nthreads().values()])\n",
    "\n",
    "def total_ncores():\n",
    "    return sum([v for v in client.ncores().values()])\n",
    "\n",
    "def total_workers():\n",
    "    return len(client.ncores())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the main loop where we time a distributed read for different numbers of worker counts.\n",
    "We also keep track of some other metadata about our testing conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "diag_kwargs = dict(nbytes=data.nbytes, chunksize=chunksize,\n",
    "                   cloud='gcloud', format='zarr')\n",
    "\n",
    "\n",
    "for nworkers in [30, 20, 10, 5]:\n",
    "    cluster.scale(nworkers)\n",
    "    time.sleep(10)\n",
    "    client.wait_for_workers(nworkers)\n",
    "    print(nworkers)\n",
    "    with diag_timer.time(nthreads=total_nthreads(),\n",
    "                         ncores=total_ncores(),\n",
    "                         nworkers=total_workers(),\n",
    "                         **diag_kwargs):\n",
    "        \n",
    "        future = dsa.store(data, null_store, lock=False, compute=False)\n",
    "        dask.compute(future, retries=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.close()\n",
    "cluster.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = diag_timer.dataframe()\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot(x='ncores', y='runtime', marker='o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['throughput_MBps'] = df.nbytes / 1e6 / df['runtime']\n",
    "df.plot(x='ncores', y='throughput_MBps', marker='o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['throughput_MBps_per_thread'] = df.throughput_MBps / df.nthreads\n",
    "df.plot(x='nthreads', y='throughput_MBps_per_thread', marker='o')\n",
    "plt.ylim([0, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Many Results\n",
    "\n",
    "Similar tests to the one above were performed with many different cloud storage technologies and data formats:\n",
    "\n",
    "- Google Cloud Storage + Zarr Format (like the example above)\n",
    "- Google Cloud Storage + NetCDF Format (NetCDF files were opened directly from object storage using `gcsfs` to provide file-like objects to `h5py`)\n",
    "- [Wasabi Cloud](https://wasabi.com/) + Zarr Format (A discount cloud-storage vendor)\n",
    "- [Jetstream Cloud](http://jetstream-cloud.org/) + Zarr Format (NSF's Cloud)\n",
    "- [Open Storage Network](https://www.openstoragenetwork.org/) (OSN) NCSA Storage Pod + Zarr Format\n",
    "- The [NASA ECCO Data Portal](https://data.nas.nasa.gov/ecco) (data read using [`xmitgcm.llcreader`](https://xmitgcm.readthedocs.io/en/latest/llcreader.html))\n",
    "- Various OPeNDAP Servers, including:\n",
    "  - The [UCAR ESGF Node](https://esgf-data.ucar.edu/thredds/catalog/catalog.html) TDS Server\n",
    "  - [NOAA ESRL PSL](https://psl.noaa.gov/thredds/catalog.html) TDS Server\n",
    "\n",
    "The results have been archived on Zenodo:\n",
    "[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3829032.svg)](https://doi.org/10.5281/zenodo.3829032)\n",
    "\n",
    "Despite the different sources and formats, all tests were conducted in the same basic manner:\n",
    "1. A \"lazy\" dask array referencing the source data was created\n",
    "2. A dask cluster was scaled up to different sizes\n",
    "3. The array was \"stored\" into the `NullStore`, forcing all data to read from the source\n",
    "\n",
    "_Note:_ All computing was done in Google Cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_base = 'https://zenodo.org/record/3829032/files/'\n",
    "files = {'GCS Zarr': ['gcloud-test-1.csv', 'gcloud-test-3.csv', 'gcloud-test-4.csv',\n",
    "                      'gcloud-test-5.csv', 'gcloud-test-6.csv'],\n",
    "         'GCS NetCDF': ['netcdf-test-1.csv'],\n",
    "         'Wasabi Zarr': ['wasabi-test-1.csv', 'wasabi-test-2.csv', 'wasabi-test-3.csv'],\n",
    "         'Jetstream Zarr': ['jetstream-1.csv', 'jetstream-2.csv'],\n",
    "         'ECCO Data Portal': ['ecco-data-portal-1.csv', 'ecco-data-portal-2.csv'],\n",
    "         'ESGF UCAR OPeNDAP': ['esgf-ucar-1.csv', 'esgf-ucar-2.csv'],\n",
    "         'NOAA ESRL OPeNDAP': ['noaa-esrl-1.csv', 'noaa-esrl-2.csv'],\n",
    "         'OSN Zarr': ['OSN-test-1.csv', 'OSN-test-2.csv']\n",
    "        }\n",
    "\n",
    "data = {}\n",
    "for name, fnames in files.items():\n",
    "    data[name] = pd.concat([pd.read_csv(f'{url_base}/{fname}') for fname in fnames])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['OSN Zarr']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig0, ax0 = plt.subplots()\n",
    "fig1, axs = plt.subplots(nrows=2, ncols=4, figsize=(20, 8))\n",
    "palette = sns.color_palette('colorblind', len(files))\n",
    "\n",
    "for (name, df), color, ax1 in zip(data.items(), palette, axs.flat):\n",
    "    for ax in [ax0, ax1]:\n",
    "        df.plot(kind='scatter', x='ncores', y='throughput_MBps',\n",
    "                  s=50, c=[color], edgecolor='k', ax=ax, label=name)\n",
    "    ax1.grid()\n",
    "    ax1.set_title(name)\n",
    "\n",
    "ax0.grid()\n",
    "ax0.legend(loc='upper left')\n",
    "\n",
    "fig0.tight_layout()\n",
    "fig1.tight_layout()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "The figure above shows a wide range of throughput rates and scaling behavior.\n",
    "There are a great many factors which might influence the results, including:\n",
    "- The structure of the source data (e.g. data format, chunk size, compression, etc.)\n",
    "- The rate at which the storage service can read and process data from its own disks\n",
    "- The transport protocol and encoding (compression, etc.)\n",
    "- The network bandwidth between the source and Google Cloud US-CENTRAL1 region\n",
    "- The overhead of the dask cluster in computing many tasks simultaneously\n",
    "\n",
    "The comparison is far from \"fair\": it is clearly biased towards Google Cloud Storage, which has optimum network proximity to the compute location. Nevertheless, we make the following observations.\n",
    "\n",
    "### Zarr is about 10x faster than NetCDF in Cloud Object Storage\n",
    "\n",
    "Using 40 cores (20 dask workers), we were able to pull netCDF data from Google Cloud Storage at a rate of about 500 MB/s.\n",
    "Using the Zarr format, we could get to 5000 MB/s (5 GB/s) for the same number of dask workers.\n",
    "\n",
    "### Google Cloud Storage Scales\n",
    "\n",
    "Both formats, however, showed good scaling, with throughput increasingly nearly linearly with the size of the cluster.\n",
    "The largest experiment we attempted used nearly 400 cores: we were able to pull data from GCS at roughly 20 GB/s.\n",
    "That's pretty fast! At this rate, we could process a Petebyte of data in about 14 hours.\n",
    "However, there is a knee in the GCS-Zarr curve around 150 cores, with the scaling becoming slightly poorer above that value.\n",
    "We hypothesize that this is due to Dask's overhead from handling a very large task graph.\n",
    "\n",
    "### External Cloud Storage Providers Scale...then Saturate\n",
    "\n",
    "We tested three cloud storage providers _outside_ of Google Cloud: Wasabi (a commercial service), Jetstream (an NSF-operated cloud), and OSN (an NSF-sponsored storage provider).\n",
    "These curves all show a similar pattern: good scaling for low worker counts, then a plateau.\n",
    "We interpret this a saturation of the network bandwidth between the data and the compute location.\n",
    "Of these three, Jetstream saturated at the lowest value (2 GB/s), then Wasabi (3.5 GB/s, but possibly not fully saturated yet), then OSN (5.5 GB/s).\n",
    "Also noteworthy is the fact that, for smaller core counts, OSN was actually _faster_ than GCS _within Google Cloud_.\n",
    "These results are likely highly sensitive to network topology.\n",
    "However, they show unambiguously that OSN is an attractive choice for cloud-style storage and on-demand big-data processing.\n",
    "\n",
    "### OPeNDAP is Slow\n",
    "\n",
    "The fastest we could get data out of an OPeNDAP server was 100 MB/s (UCAR ESGF Node).\n",
    "The NOAA ESRL server was more like 20 MB/s.\n",
    "These rates are many orders of magnitude than cloud storage.\n",
    "Perhaps these servers have throttling in place to limit their total bandwidth.\n",
    "Or perhaps the OPeNDAP protocol itself has some fundamental inefficiencies.\n",
    "OPeNDAP remains a very convenient protocol for remote data access; if we wish to use it for Big Data and distributed processing, data providers need to find some way to speed it up.\n",
    "\n",
    "## Conclusions\n",
    "\n",
    "We demonstrated a protocol for testing the throughput and scalability Array data storage with the Dask computing framework.\n",
    "Using Google Cloud as our compute platform, we found, unsurprisingly, that a cloud-optimized storage format (Zarr), used with in-region cloud storage, provided the best raw throughput (up to 20 GB/s in our test) and scalability.\n",
    "We found that the same storage format also allowed us to transfer data on-demand from other clouds at reasonable rates (5 GB/s) for modest levels of parallelism.\n",
    "(Care must be taken in this case, however, because many cloud providers charge egress fees for data leaving their network.)\n",
    "Combined with Zarr, all of the cloud object storage platforms were able to deliver data orders of magnitude faster than traditional OPeNDAP servers, the standard remote-data-access protocol for the geosciences.\n",
    "\n",
    "### Future Work\n",
    "\n",
    "This analysis could be improved in numerous ways. In the future we would like to\n",
    "- Investigate other cloud-native storage formats, such as TileDB and Cloud-Optimized Geotiff\n",
    "- Perform deeper analysis of parameter sensitivity (chunk size, compression, etc.)\n",
    "- Explore other clouds (e.g. AWS, Microstoft Azure)\n",
    "\n",
    "Finally, we did not attempt to assess the relative costs associated with cloud vs. on-premises data storage, as this involves difficult questions related to total cost of ownership.\n",
    "However, we hope these results are useful for data providers in deciding how to provision future infrastructure for data-intensive geoscience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Reviews\n",
    "\n",
    "Here we include the review of the original version of this notebook and our replies.\n",
    "\n",
    "### Review 1\n",
    " \n",
    "#### Evaluation of the Contribution\n",
    "Overall Recommendation (100%): 9\n",
    "Total points (out of 10)    : 9\n",
    "\n",
    "\n",
    "#### Comments for the Authors\n",
    "\"Big Arrays, Fast: Profiling Cloud Storage Read Throughput\" is a good quality\n",
    "and, most of all, interesting science meets computer science notebook. It\n",
    "addresses important issues relevant to this scientific arena. The problem\n",
    "statement, discussion and conclusions are all compelling and pertinent to this\n",
    "area of inquiry.\n",
    "\n",
    "The most important criterion \"does it run\" is satisfied as I was able to run\n",
    "the notebook to completion on https://ocean.pangeo.io (though I could not run\n",
    "it on Binder for some reason, \"KeyError: 'GFDL_CM2_6_control_ocean'\").\n",
    "\n",
    "> The error has been corrected in the current version.\n",
    "\n",
    "I especially liked this notebook for the following reason:\n",
    "\n",
    "- The \"computational essay\" aspect receives high marks for having expository\n",
    "prose and computation work together to build a logical conclusion.\n",
    "- Figures are generated inline with matplotlib\n",
    "- Figures are easy to understand and interesting (e.g., the set of throughput\n",
    "figures at the end of the notebook).\n",
    "- The xarray.Dataset meta data allows for inspection via the attributes and\n",
    "data representation icons.\n",
    "- Computational cells are small and intermixed with expository prose.\n",
    "\n",
    "Problems with this notebook:\n",
    "\n",
    "- Quite a few distracting spelling errors: Benachmarks, analagous, simultaneouslly, slighly, saturdation\n",
    "\n",
    "> Unfortunately, JupyterLab has no spell checker, a feature I have obviously become to reliant upon.\n",
    "\n",
    "A suggestion for improvement would be to include a reference section with HTML\n",
    "linked DOIs for further inquiry and background material, and perhaps also a\n",
    "future work section.\n",
    "\n",
    "> A future work section has been included.\n",
    "\n",
    "\n",
    "### Review 2\n",
    "\n",
    "#### Evaluation of the Contribution\n",
    "\n",
    "Overall Recommendation (100%): 8\n",
    "Total points (out of 10)    : 8\n",
    "\n",
    "\n",
    "#### Comments for the Authors\n",
    "\n",
    "- the author should add to the discussion a reason for the choice of threads,\n",
    "cores and workers in the 'do benchmarking' section\n",
    "\n",
    "> This has been done\n",
    "\n",
    "- while the analysis is designed to provide a coarse grain view of the speed\n",
    "of zarr, it might be a valuable exercise to break the scales down to segments\n",
    "where they can be compared; e.g. of the four zarr tests, the n_cores scales\n",
    "vary from 0..400 (gcs), 0..60 (wasabi, jetstream) and 0..175 (osn) and thus\n",
    "these scales make it difficult to compare them when we do not have data past\n",
    "n_cores=60 and again n_cores=175\n",
    "\n",
    "> This a reasonable suggestion but beyond what we could accomplish in a brief revision period.\n",
    "\n",
    "- though the authors acknowledge the tests are not 'fair', indeed it would be\n",
    "more valuable to compare various configurations of netcdf to make such\n",
    "comparisons more easily interpretable\n",
    "\n",
    "> It's not clear what other \"configurations\" of netCDF are being suggested here. \n",
    "\n",
    "- typo: change to \"saturation\" > We interpret this a saturdation of the\n",
    "network bandwidth between the data and the compute location.\n",
    "\n",
    "> Thanks, done.\n",
    "\n",
    "- the abstract should include \"preliminary\" to situate this work as\n",
    "in-progress and not a full quantitative exploration and assessment of\n",
    "accessing data\n",
    "\n",
    "> Given the combinatoric explosion of possible configuration choices, is a comprehensive \"full quantitative exploration\" even possible?\n",
    "\n",
    "- if at all possible the author should provide instructions for how to execute\n",
    "elsewhere outside the pangeo platform\n",
    "\n",
    "\n",
    "> We have included some new links to how to deploy dask in other environments.\n",
    "\n",
    "- licensing information as well as notebook requirements should also be\n",
    "provided with the notebook (or a link to the GH repo if one exists for the\n",
    "notebook)\n",
    "\n",
    "> This was all included in the original submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}