COAWST_explore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3a3edc9f-3c66-4e03-8c75-b147f15b2e50",
   "metadata": {},
   "source": [
    "# Explore the COAWST US East Coast and Gulf of Mexico Forecast Archive Dataset\n",
    "This is a cloud-optimized version of the NetCDF files accessed from the USGS ScienceBase item [Collection of COAWST model forecast for the US East Coast and Gulf of Mexico](https://www.sciencebase.gov/catalog/item/610acd4fd34ef8d7056893da).   The original daily forecast files were converted into weekly NetCDF files with 168 points in the time dimension to facilitate time series access. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "547b851f-3dd9-4b9f-988b-54c743a43bbe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import fsspec\n",
    "import xarray as xr\n",
    "import hvplot.xarray\n",
    "import intake\n",
    "import cf_xarray\n",
    "import numpy as np\n",
    "import panel as pn\n",
    "from matplotlib import path\n",
    "import xoak"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57354df2-6786-4d1d-859e-1d5099cb85b9",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Open Dataset\n",
    "\n",
    "The details of data loading are stored in an `intake` catalog, which simplifies use.  Metadata and coordinate data are loaded, but not the actual data variables, which are loaded only as needed by subsequent analysis and visualization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32cf5f03-3461-4548-b911-0ce7c8970e6a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "intake_catalog_url = 'https://usgs-coawst.s3.amazonaws.com/useast-archive/coawst_intake.yml'\n",
    "cat = intake.open_catalog(intake_catalog_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a233306-2b00-40d9-aa64-79449c993797",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6d6c625-b5d7-4a17-ae2d-d7c2e8bc3fa6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset = 'COAWST-USEAST' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5320108-9074-40b3-a5e5-e3457af690ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cat[dataset]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59028b58-1a1a-4312-b8e1-ce11200eefef",
   "metadata": {},
   "source": [
    "This is a big dataset, so it takes up to 30s to open the dataset (which involves reading all the metadata and index coordinate variable data). Here we load the data into xarray using `.to_dask()` so that if we have a Dask cluster, we can speed up data processing by loading and processing chunks of data in parallel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3b2ce8a-6499-4413-861c-65d48f7af108",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "ds = cat[dataset].to_dask()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cfe57c8-3892-4a02-9683-a7814e266b5a",
   "metadata": {},
   "source": [
    "Let's look at that metadata.  We can explore the different attributes and variables by clicking on the variables and icons below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "608b1398-bf35-40c8-bd9f-ff21238ae591",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds.nbytes/1e12"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15db8c13-9961-4be1-ab57-382d75f30999",
   "metadata": {},
   "source": [
    "We can also explore a specific variable of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c72e31c-210a-4025-8293-478bb0f38b87",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "var = 'Hwave'\n",
    "da = ds[var]\n",
    "da"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd304dcd-079f-498a-bff6-d27cd9a118a7",
   "metadata": {},
   "source": [
    "Use the CF conventions to identify the coordinate variables for longitude, latitude and time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90ff7b29-0c6f-4fa4-aa01-d10e2ee0e00a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = da.cf['longitude']\n",
    "y = da.cf['latitude']\n",
    "t = da.cf['time']\n",
    "print(x.name, y.name, t.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e736ff3-b0e4-4452-af46-1afa462af185",
   "metadata": {},
   "source": [
    "## Example: Load the entire spatial domain for a variable at a specific time step\n",
    "Loading the entire spatial domain at a time step only requires reading 8 chunks of data, so it loads in a few seconds.  A dask cluster doesn't help much in this case as it's already fast.   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21853984-cc84-4c38-80cf-4349c7fe5787",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "da2d = da.cf.sel(T='2012-10-29 12:00', method='nearest').load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f46edc95-8904-4256-9729-7c09f03c51d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da2d.hvplot.quadmesh(x=x.name, y=y.name, rasterize=True, geo=True, tiles='OSM', cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c71dae-3e8e-4293-b741-9c0992cc4c52",
   "metadata": {},
   "source": [
    "## Example: Load a time series for a variable at a specific lon,lat location for a specified time range. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd35652-608d-4eca-94d9-78cfd3591b38",
   "metadata": {},
   "source": [
    "To identify a point, we will start with its lat/lon coordinates.  If lon and lat were 1D coordinates, we could use lon,lat values to select using xarray, but instead we need to extract using indices, which we need to find.   For this we use the `xoak` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e1bae3e-54dc-46fd-8d52-c8bd48719285",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "lat,lon = 42.5, -70.0  # Gulf of Maine, 100km east of Boston, MA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "654b797b-5720-49ac-8376-9f64a9113bb4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da.xoak.set_index([y.name, x.name], 'scipy_kdtree')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c13c1276-1390-4ab3-ae64-bf77fae19327",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_point = xr.Dataset({\"lon\": (\"point\", [lon]), \"lat\": (\"point\", [lat])})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a7a4047-2104-423b-bdae-edf1a623c3d8",
   "metadata": {},
   "source": [
    "Before we read the data, let's see how many chunks we will be reading:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f38d8247-2b9b-4ded-a5b5-76c4bd603573",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da.xoak.sel(lat_rho=ds_point.lat, lon_rho=ds_point.lon).cf.sel(T='2012-10')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a2e75a7-a734-4f1d-bea9-1db198463326",
   "metadata": {},
   "source": [
    "To load this one month means reading 5 chunks of data, so still don't need a cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38db110b-954f-4978-9797-610afba72549",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    " da.xoak.sel(lat_rho=ds_point.lat, lon_rho=ds_point.lon).cf.sel(T='2012-10')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53bbf395-9ea9-4ccb-810e-42679738015f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "da1d = da.xoak.sel(lat_rho=ds_point.lat, lon_rho=ds_point.lon).cf.sel(T='2012-10').load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44fbef48-c3f7-4c07-bed8-2aff43e92c0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da1d.hvplot(x=t.name, grid=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d7d72a8-a012-43da-aba3-5826d18f7db0",
   "metadata": {},
   "source": [
    "How many chunks of data will we read to load the entire time series of record at a point?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9eba147-c25a-48e6-9b57-98d2c243dfe9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da.xoak.sel(lat_rho=ds_point.lat, lon_rho=ds_point.lon)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c634ac99-e2d8-448f-bedc-95021124f859",
   "metadata": {},
   "source": [
    "Since we now need to read 669 chunks of data, we should use a Dask cluster if we have access to one"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecef85f6-01e4-4d84-9cce-0bf409b30637",
   "metadata": {},
   "source": [
    "### Parallelize with Dask \n",
    "We opened the dataset so that we can take advantage of parallel compute environments\n",
    "using `dask`. We're going to start a cluster now so that future steps can take advantage\n",
    "of this ability. \n",
    "\n",
    "This is an optional step, but speeds up data loading and processing significantly, especially \n",
    "when accessing data from the cloud.\n",
    "\n",
    "There are many ways to [deploy a Dask cluster](https://docs.dask.org/en/stable/deploying.html#deploy-dask-clusters).   \n",
    "Below each cell uses a different approach.   Use one of the approaches below or choose another method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df2761d2-49ae-4204-a0f9-5fedee74f977",
   "metadata": {},
   "outputs": [],
   "source": [
    "#cluster_type = 'Local'    \n",
    "#cluster_type = 'Coiled'\n",
    "# cluster_type = 'Gateway'\n",
    "cluster_type = 'Coiled'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd49f583-413e-43fa-bd7d-c0de037565b9",
   "metadata": {},
   "source": [
    "#### Use LocalCluster\n",
    "LocalCluster is available in any computing environment.  It uses the number of CPUs of the computer running the notebook to create a cluster. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89e495ca-ab99-468e-801f-b3ca494f3a19",
   "metadata": {},
   "outputs": [],
   "source": [
    "if cluster_type == 'Local':\n",
    "    from dask.distributed import LocalCluster, Client\n",
    "    cluster = LocalCluster()\n",
    "    client = Client(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bba126c-aa5e-42b2-a6c3-a319c9382065",
   "metadata": {},
   "source": [
    "#### Use Coiled\n",
    "[Coiled](https://www.coiled.io/) provides access to remote Dask clusters that can be used from anywhere.  It requires a Coiled account. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eaf2c780-d76d-4c6f-8702-a7fdbe14daec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if cluster_type == 'Coiled':\n",
    "    import coiled\n",
    "    cluster = coiled.Cluster(\n",
    "        region=\"us-west-2\",\n",
    "        arm=True,   # run on ARM to save energy & cost\n",
    "        worker_vm_types=[\"t4g.small\"],  # cheap, small ARM instances, 2cpus, 2GB RAM\n",
    "        worker_options={'nthreads':2},\n",
    "        n_workers=30,\n",
    "        wait_for_workers=False,\n",
    "        compute_purchase_option=\"spot_with_fallback\",\n",
    "        name='coawst',   # Dask cluster name\n",
    "        software='esip-pangeo-arm',  # Conda environment name\n",
    "        workspace='esip-lab',\n",
    "        timeout=180   # leave cluster running for 3 min in case we want to use it again\n",
    "    )\n",
    "\n",
    "    client = cluster.get_client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b75863b6-7dfc-4739-a0a4-4c2e7daea039",
   "metadata": {},
   "source": [
    "#### Use a Dask Gateway Cluster\n",
    "[Dask Gateway](https://gateway.dask.org/) is a common way to spin up a Dask Cluster.  [Nebari](https://nebari.dev) and [DaskHub](https://github.com/dask/helm-chart) are popular ways of deploying a JupyterHub with Dask Gateway.  You can use a JupyterHub with DaskGateway for free by [signing up for access to the Microsoft Planetary Computer hub](https://planetarycomputer.microsoft.com/account/request)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddbb44f9-00f8-44c4-8a9c-d9d09128361b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "if cluster_type == 'Gateway':\n",
    "    from dask_gateway import Gateway\n",
    "\n",
    "    gateway = Gateway()  # instantiate Dask gateway \n",
    "\n",
    "    # Cluster options on Nebari \n",
    "    options = gateway.cluster_options()\n",
    "    options.conda_environment='global/global-pangeo'  # comment out for Daskhub or Planetary Computer\n",
    "    options.profile = 'Small Worker'   # comment out for Daskhub or Planetary Computer\n",
    "\n",
    "    # Create a Dask Gateway cluster\n",
    "    cluster = gateway.new_cluster(options)\n",
    "\n",
    "    # Get the Dask client for the Dask Gateway cluster\n",
    "    client = cluster.get_client()\n",
    "\n",
    "    # Scale the cluster\n",
    "    cluster.adapt(minimum=4, maximum=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5312240a-e959-467e-84f2-3482a1cb52ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "if cluster_type == 'Nebari':\n",
    "    import sys, os\n",
    "    sys.path.append(os.path.join(os.environ['HOME'],'shared','users','lib'))\n",
    "    import nebari_tools as nbt\n",
    "\n",
    "    aws_profile = 'esip-qhub'\n",
    "    aws_region = 'us-west-2'\n",
    "    endpoint_url = f's3.{aws_region}.amazonaws.com'\n",
    "\n",
    "    nbt.set_credentials(profile=aws_profile, region=aws_region, endpoint_url=endpoint_url)\n",
    "    worker_max = 30\n",
    "\n",
    "    client, cluster = nbt.start_dask_cluster(profile=aws_profile, worker_max=worker_max, \n",
    "                                          region=aws_region, use_existing_cluster=True,\n",
    "                                          adaptive_scaling=True, wait_for_cluster=True, \n",
    "                                          worker_profile='Small Worker', \n",
    "                                          propagate_env=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e53730ec-ace1-4994-8405-9becf6f8380a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f45e3c27-2002-488a-b185-146df1307054",
   "metadata": {},
   "source": [
    "Load the entire time series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4695724b-dbaf-4d2a-9343-fa0077ccbd58",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "ds_selection = da.xoak.sel(lat_rho=ds_point.lat, lon_rho=ds_point.lon).load()       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b36b3b6f-f815-4576-be1e-6aac40210abe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_selection.hvplot(x=t.name, grid=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0472ff9-ec80-43c3-8fb2-2d096415b002",
   "metadata": {},
   "source": [
    "## Example: Compute the time mean for a variable over the entire domain for a specific time period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e094faaf-fd3f-45eb-a65c-c4c1adaf3c33",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "da_mean = da.cf.sel(T=slice('2016-01-01 00:00','2017-01-01 00:00')).mean(dim=t.name).compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b51ab94-1776-421f-9b93-c02c2d5da319",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "da_mean.hvplot.quadmesh(x=x.name, y=y.name, rasterize=True, geo=True, tiles='OSM', cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c415fcc6-fa98-44f1-8e17-99034f717cb8",
   "metadata": {},
   "source": [
    "## Example: Subset a time and space region and export to NetCDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa229688-91ef-4337-a051-71299b3c17a0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def bbox2ij(lon,lat,bbox=[-160., -155., 18., 23.]):\n",
    "    \"\"\"Return indices for i,j that will completely cover the specified bounding box.     \n",
    "    i0,i1,j0,j1 = bbox2ij(lon,lat,bbox)\n",
    "    lon,lat = 2D arrays that are the target of the subset\n",
    "    bbox = list containing the bounding box: [lon_min, lon_max, lat_min, lat_max]\n",
    "\n",
    "    Example\n",
    "    -------  \n",
    "    >>> i0,i1,j0,j1 = bbox2ij(lon_rho,lat_rho,[-71, -63., 39., 46])\n",
    "    >>> h_subset = nc.variables['h'][j0:j1,i0:i1]       \n",
    "    \"\"\"\n",
    "    bbox=np.array(bbox)\n",
    "    mypath=np.array([bbox[[0,1,1,0]],bbox[[2,2,3,3]]]).T\n",
    "    p = path.Path(mypath)\n",
    "    points = np.vstack((lon.ravel(),lat.ravel())).T   \n",
    "    n,m = np.shape(lon)\n",
    "    inside = p.contains_points(points).reshape((n,m))\n",
    "    ii,jj = np.meshgrid(range(m),range(n))\n",
    "    return min(ii[inside]),max(ii[inside]),min(jj[inside]),max(jj[inside])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85b4ba2a-1385-4002-b7e6-596b2b5025b4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bbox = [-76.63290610753754, -73.55671530588432, 37.57888442021855, 41.225532965406224]   # DRB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d0659c1-2757-43d6-9289-be55c4a39daf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "i0,i1,j0,j1 = bbox2ij(x.values, y.values, bbox=bbox)\n",
    "print(i0,i1,j0,j1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee6e947-afbe-4b2b-88ff-59e3f84ef239",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_drb = ds[['temp', 'salt', 'Hwave']].isel(eta_rho=slice(j0,j1), xi_rho=slice(i0,i1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8cca25f-ef67-4f0a-9cc9-a4f7d4bd1add",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_drb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f9f2eff-25c6-46ec-b735-7ec57195db41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_drb_timeslice = ds_drb.cf.sel(T=slice('2022-04-01 00:00','2022-04-08 00:00'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f35132f-3881-449e-b4e0-8178beeaacfc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds_drb_timeslice = ds_drb_timeslice.chunk({'eta_rho':-1, 'xi_rho':-1})  # chunk to full spatial subset domain\n",
    "print(f'Uncompressed dataset size: {ds_drb_timeslice.nbytes/1e6} MB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c55baf-1388-48ad-a1c4-fa13b063f1f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "var = 'salt'\n",
    "da_drb = ds_drb_timeslice[var].load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6067684-8781-43b0-aacd-308e8a32709d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "viz = da_drb.hvplot.quadmesh(x=x.name, y=y.name, geo=True,\n",
    "                    cmap='turbo', rasterize=True, tiles='OSM', title=var)\n",
    "viz = pn.panel(viz, widgets={'ocean_time': pn.widgets.Select} )\n",
    "pn.Column(viz).servable('DRB Explorer')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21824171-58f8-4677-bf2b-7ca4ec592389",
   "metadata": {},
   "source": [
    "Close the Dask client since we can't write NetCDF in parallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb417e27-333e-41f9-ba7a-157708a1ff24",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d76c3cc-96bd-4c66-a9d1-12af3dd659d2",
   "metadata": {},
   "source": [
    "Specify the encoding to enable compression in the NetCDF file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27d11d63-668f-466b-be4c-1a570f6d9f96",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "encoding={}\n",
    "for var in ds_drb_timeslice.variables:\n",
    "    encoding[var] = dict(zlib=True, complevel=4, \n",
    "                         fletcher32=False, shuffle=True,\n",
    "                         _FillValue=None)\n",
    "\n",
    "ds_drb_timeslice.to_netcdf('drb.nc', encoding=encoding, mode='w')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9230a094-4dcd-4552-852e-be62cb29e15d",
   "metadata": {},
   "source": [
    "## Stop cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b30b600-c981-4d6b-a251-14e3770c1b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.shutdown()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "global-global-pangeo",
   "language": "python",
   "name": "conda-env-global-global-pangeo-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}