05-HSDS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Highly Scalable Data Service (HSDS)\n",
    "\n",
    "<a href=\"https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/\"><img src=\"img/hsds.png\" alt=\"HSDS architecture\" /></a>\n",
    "\n",
    "## Installation\n",
    "\n",
    "For this simplistic setup, the installation is easy:\n",
    "\n",
    "1. Create a directory for the HSDS data files (`~/hsds_data`).\n",
    "2. Use the user name `vscode` and password `vscode` to authenticate to HSDS.\n",
    "3. Launch the service.\n",
    "\n",
    "For more sophisticated setups (e.g., Kubernetes), please refer to the [HSDS documentation](https://github.com/HDFGroup/hsds/tree/master/docs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export HS_ENDPOINT=http://localhost:5101\n",
    "export HS_USERNAME=$USER\n",
    "export HS_PASSWORD=$USER\n",
    "mkdir ~/hsds_data\n",
    "hsds --root_dir ~/hsds_data --hs_username $USER --hs_password $USER >~/hs.log 2>&1 &\n",
    "sleep 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a configuration file for HSDS: `~/.hscfg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsconfigure <<< $'http://localhost:5101\\nvscode\\nvscode\\n\\nY\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that the server is running and the configuration is correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the top-level domain and a user \"directory\" for the user `vscode`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hstouch /home/ && hstouch /home/$USER/\n",
    "hsinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kicking the tires\n",
    "\n",
    "Let's create a simple HDF5 domain with a single dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5pyd\n",
    "\n",
    "f = h5pyd.File(\"/home/vscode/foo.h5\", \"w\")\n",
    "dset = f.create_dataset(\"dset\", data=[1,2,3,4])\n",
    "f.id.id\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is **not** a file in the local file system, but a domain in the HSDS service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ls -l /home/vscode/foo.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where is it? Let's check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "curl -s -u vscode:vscode http://localhost:5101/?domain=/home/vscode/foo.h5 | jq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "find ~/hsds_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple command line tools are available to interact with the service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsls /home/vscode/foo.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsstat /home/vscode/foo.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is HSDS really HDF5?\n",
    "\n",
    "We can reuse the Python version of our HDF5 example with only two trivial changes:\n",
    "\n",
    "1. We use the `h5pyd` package instead of `h5py`. (See line 2.)\n",
    "2. The file name is now a domain name (i.e., `/home/vscode/ou_h5pyd.h5`). (See line 30.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import h5pyd as h5py\n",
    "\n",
    "def ou_sampler(path_count, step_count, dt, theta, mu, sigma):\n",
    "    '''\n",
    "    Generates sample paths for an Ornstein-Uhlenbeck process.\n",
    "    '''\n",
    "    ou_process = np.zeros((path_count, step_count))\n",
    "    for i in range(path_count):\n",
    "        for j in range(1, step_count):\n",
    "            dW = np.random.normal(0, np.sqrt(dt))\n",
    "            ou_process[i, j] = ou_process[i, j-1] + theta * (mu - ou_process[i, j-1]) * dt + sigma * dW\n",
    "    return ou_process\n",
    "\n",
    "def main():\n",
    "    # Parameters\n",
    "    path_count = 100\n",
    "    step_count = 1000\n",
    "    dt = 0.01\n",
    "    theta = 1.0\n",
    "    mu = 0.0\n",
    "    sigma = 0.1\n",
    "\n",
    "    print(\"Running with parameters:\", \"paths=\", path_count, \"steps=\", step_count, \"dt=\", dt, \"theta=\", theta, \"mu=\", mu, \"sigma=\", sigma)\n",
    "\n",
    "    # Generate OU process sample paths\n",
    "    ou_process = ou_sampler(path_count, step_count, dt, theta, mu, sigma)\n",
    "\n",
    "    # Write sample paths to an HDF5 file\n",
    "    with h5py.File('/home/vscode/ou_h5pyd.h5', 'w') as file:\n",
    "        file.attrs['source'] = 'https://github.com/HDFGroup/hdf5-tutorial'\n",
    "\n",
    "        # Create & write the dataset\n",
    "        dataset = file.create_dataset('dataset', data=ou_process)\n",
    "\n",
    "        # Add documentation to the dataset\n",
    "        file['dataset'].attrs['comment'] = 'This dataset contains sample paths of an Ornstein-Uhlenbeck process.'\n",
    "        file['dataset'].attrs['Wikipedia'] = 'https://en.wikipedia.org/wiki/Ornstein–Uhlenbeck_process'\n",
    "        file['dataset'].attrs['rows'] = 'path'\n",
    "        file['dataset'].attrs['columns'] = 'time'\n",
    "        \n",
    "        # Set attributes\n",
    "        file['dataset'].attrs['dt'] = dt\n",
    "        file['dataset'].attrs['θ'] = theta\n",
    "        file['dataset'].attrs['μ'] = mu\n",
    "        file['dataset'].attrs['σ'] = sigma\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsls --showattrs -r /home/vscode/ou_h5pyd.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization of the data with HSDS is identical with the same two trivial changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import h5pyd as h5py\n",
    "import numpy as np\n",
    "\n",
    "f = h5py.File(\"/home/vscode/ou_h5pyd.h5\")\n",
    "dset = f[\"dataset\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = dset[42,:]\n",
    "print(f\"min: {arr.min():.2f}, max: {arr.max():.2f}, mean: {arr.mean():.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.style.use('_mpl-gallery')\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(np.arange(0,len(arr)), arr, linewidth=2.0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HSDS and HDF5 files\n",
    "\n",
    "It is easy to import and export HDF5 files from and to HSDS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsload ou_process.h5 /home/vscode/ou_process.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsstat /home/vscode/ou_process.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsls --showattrs -r /home/vscode/ou_process.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Export an HSDS domain to an HDF5 file and test for equality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsget /home/vscode/ou_process.h5 ou_process_copy.h5\n",
    "h5diff ou_process.h5 ou_process_copy.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "HSDS is a highly scalable data service for HDF5 data. It is easy to install and use. It is a great tool for sharing HDF5 data with others.\n",
    "\n",
    "HSDS implements the HDF5 data model and, as we will see in a moment, is compatible with the HDF5 API.\n",
    "\n",
    "HSDS complements the HDF5 library and file format. It is not a replacement, but a great addition to the HDF5 ecosystem and the better option for many use cases in the cloud, and, generally, where a service-oriented interface is a better fit."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hdf5-tutorial",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}