docs/source/tutorials/map-options.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Map Options\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/htcondor/htmap/master?urlpath=lab%2Ftree%2Fmap-options.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requesting Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most common kind of map option you'll probably need to work with are the ones for requests resources.\n",
    "HTMap makes fairly conservative default choices about the resources required by your map components.\n",
    "If your function needs a lot of resources, such as memory or disk space, you will need to communicate this to HTMap.\n",
    "\n",
    "Suppose we need to transfer a huge input file that we need to read into memory, so we need a lot of memory and disk space available on the execute node.\n",
    "We'll request 200 MB of RAM, 10 GB of disk space, and send our input file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import htmap\n",
    "\n",
    "def read_huge_file(file):\n",
    "    contents = Path(file).read_text()\n",
    "\n",
    "    # do stuff\n",
    "\n",
    "    return contents  # we'll just return the contents here, but imagine this is the result of processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "huge_file = htmap.TransferPath.cwd() / 'huge_file.txt'\n",
    "huge_file.write_text('only a few words, but use your imagination')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Don't panic! `write_text()` returns the number of bytes written.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's our map call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created map breezy-thick-beak with 1 components\n",
      "only a few words, but use your imagination\n"
     ]
    }
   ],
   "source": [
    "processed = htmap.map(\n",
    "    read_huge_file,\n",
    "    [huge_file],\n",
    "    map_options = htmap.MapOptions(\n",
    "        request_memory = '100MB',\n",
    "        request_disk = '1GB',\n",
    "    ),\n",
    ")\n",
    "\n",
    "print(processed.get(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`request_memory` and `request_disk` were passed as single strings.\n",
    "Since they are single strings, they will be treated as **fixed options** and applied to every component.\n",
    "The other kind of option is **variadic**, which lets you specify some option for each component of the map individually.\n",
    "For exampe, if we wanted a different amount of RAM for each component, we could pass a list of strings to `request_memory`, one for each component:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created map tall-soft-stream with 3 components\n",
      "['only a few words, but use your imagination', 'only a few words, but use your imagination', 'only a few words, but use your imagination']\n"
     ]
    }
   ],
   "source": [
    "multiple = htmap.map(\n",
    "    read_huge_file,\n",
    "    [huge_file, huge_file, huge_file],\n",
    "    map_options = htmap.MapOptions(\n",
    "        request_memory = ['10MB', '20MB', '30MB'],\n",
    "        request_disk = '1GB',\n",
    "    ),\n",
    ")\n",
    "print(list(multiple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Kitchen Sink"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTMap also supports **arbitrary HTCondor submit descriptors**, like you would see in a [submit file](http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html).\n",
    "Just pass them as keyword arguments to a [htmap.MapOptions](https://htmap.readthedocs.io/en/stable/api.html#htmap.MapOptions), keeping in mind that you can use standard ClassAd interpolation and that the same fixed/variadic behavior applies.\n",
    "\n",
    "If that didn't make sense, don't worry about it!\n",
    "The whole point of HTMap is to avoid needing to know too much about submit descriptors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "The [next tutorial](advanced-mapping.ipynb) discusses more convenient and flexible way of defining your maps."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}