01_dsvm_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DSVM setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the setup done in this notebook is done either through the Azure portal (https://portal.azure.com) or at the command line.  I use SSH for connecting to the remote. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select a VM type\n",
    "I used an Azure NC6 instance that had fastai installed, and later updated the OS from Ubuntu 16.04 to Ubuntu 18.04 (stable).  If you're using Amazon AWS instead, fastai may have a pre-built machine image under \"Community AMIs\" (they did in the past, but I haven't checked recently).\n",
    "\n",
    "## Add SSH\n",
    "Optional, but it makes life easier and more secure.  \n",
    "Steps:\n",
    "* Generate a public/private SSH key pair, if you don't have one already.  See XXX\n",
    "* Copy your ssh public key over to the server:\n",
    "```bash\n",
    "ssh-copy-id -i ~/.ssh/<my_public_key>.pub <user>@<url> #e.g. john@12.34.567.89\n",
    "```\n",
    "* Turn off password access  ...\n",
    "* See https://medium.com/@drjohnpayne/graduating-from-one-vm-to-two-f4084f1c331d for a more involved process if you are trying to run Jupyter Lab on multiple VMs through SSH (includes a required networking tweak)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Editor of your choice\n",
    "I use Vim with [tmux](https://github.com/tmux/tmux/wiki) because Vim is universally available and tmux enables a windowed terminal and runs a tiny server that remembers the state of your sessions.  But whatever you like is fine. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add or mount a data disk\n",
    "The default OS disk for an NC6 VM is 150 Gb, which is too small for holding lots of data. Perhaps I am confused but I think that Azure DSVM images change frequently and I think I've used DSVM's that initially only had a temporary data disk attached, which would vanish like the morning dew when the machine was shut down.  I think some DSVMs also came with data disks _that were present, pre-formatted, and already had file systems, but were not mounted nor visible in the portal_, which is odd.  Anyway, you may need to add a disk.  \n",
    "\n",
    "#### Step 1. Check to see if you already have a data disk\n",
    "```bash\n",
    "sudo fdisk -l #list disks.  Look for a big one (probably 1Tb or larger)\n",
    "```\n",
    "If you do, then skip Step 2.\n",
    "\n",
    "#### Step 2.  Initialize, partition, and format the new disk\n",
    "* Various drive types can be used; SSD standard is fast; SSD premium has much higher I/O rates but beyond what I need.\n",
    "* Use the Azure web page for managing the VM (while it is shut down) to add a new disk.  Select size and “standard” and name it.\n",
    "\n",
    "Note: az login takes you to a browser window and sends a code you have to enter in the Terminal window, to get access.\n",
    "```bash\n",
    "ssh <user@12.34.567.89>\n",
    "az login \n",
    "```\n",
    "Look at the existing attached disks to see what their names are. This doesn’t give you much information, so once you have guessed which one you want (typically the last in the “sd_” series, although they may not be in order), you can double-check it:\n",
    "```bash\n",
    "dmesg | grep SCSI \n",
    "#Check to make sure it is the right disk (l = list).  Here, the disk I wanted was “sdd”.\n",
    "sudo fdisk -l /dev/sdd   \n",
    "sudo fdisk /dev/sdd #start the fdisk program (interactive). \n",
    "    - n #create new partition\n",
    "    - p #primary\n",
    "    - <Enter> (2-3 times — just accept defaults)\n",
    "    - o #print the new partition table (just for seeing how it looks)\n",
    "    - w #write it (exits when done)\n",
    "#Update the kernel (whatever the fuck that means—maybe force the Linux kernel to re-check its parts?)\n",
    "partprobe \n",
    "#Create a file system.  WARNING: make sure you have the right drive.  The final ‘1’ is the ‘first partition’, i.e., the one you just created.\n",
    "sudo mkfs -t ext4 /dev/sdd1 \n",
    "```\n",
    "#### Step 3. Mount the disk\n",
    "```bash\n",
    "sudo mkdir /cdata #Make a directory for the data drive\n",
    "#WARNING: make sure you’re mounting the right drive (sdd1) to the right directory (/cdata)\n",
    "sudo mount /dev/sdd1 /cdata \n",
    "```\n",
    "NOTE: The default mount position in Ubuntu would be `/media/user/drive`, e.g. `/media/<user>/cdata`.  That's probably better than `/cdata` in the long run (but don’t forget to also change the line in fstab (next) if you do that.\n",
    "\n",
    "#### Step 4. Add the drive to `fstab` so it is mounted automatically when the computer boots\n",
    "Add the drive to fstab, which runs whenever the computer is re-booted.  We use the unique drive identifier (UUID) to make sure that we don’t swap drives by mistake.  \n",
    "```bash\n",
    "sudo blkid #Record the UUID for the next step\n",
    "sudo vim /etc/fstab #SERIOUS WARNING: mis-editing fstab can kill your whole machine!\n",
    "#Add this to the last line (use the UUID from the blkid step):\n",
    "UUID=f08b1c62-9698-4597-b739-e715a0bd7e1d   /cdata   ext4   defaults,nofail   1   2\n",
    "#If you have ownership problems, then change ownership of the drive    \n",
    "sudo chown -R egdod /cdata\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install additional deep learning libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fastai\n",
    "Among other packages, I use the popular FastAI library (see https://fast.ai).  The version 2 code is currently feature-complete and relatively stable.  The main benefits of the fastai library are:\n",
    "1. It has good high-level functions for munging data\n",
    "2. It has optimized versions of many PyTorch and/or Python functions.  For example, all batch-wise image augmentations are calculated together, i.e., fastai calculates in advance what the fate of each pixel will be after all augmentations are applied and then applies the final changes _once_, on the GPU, which makes augmentation run extremely fast;\n",
    "3. It has a _superb_ callback-based system for modifying and customizing the training process and training cycles; nothing else I've seen comes close.   \n",
    "4. It's based on PyTorch, which is currently an industry standard, and it has a huge, active user community.\n",
    "\n",
    "However, the fastai library is a constantly moving target.  There are currently two full versions of the code (Google searches tend to take you to the older one, which is called fastai1), and you want the newer version.  There is a third version in development.  My best advice is to follow the installation advice of the day on [fastai github site](https://github.com/fastai/fastai)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other libraries that make life better\n",
    "- **jupyterlab** FastAI is set up for Jupyter Notebook, but that only allows you to have one notebook open at a time, which is a really painful limitation.  Jupyter Lab is built on the same foundations but allows you to have a much more wonderful setup with multiple tabs in a window, each for a different notebook, and each running a different kernel (for example, 4 notebooks could simultaneously run Python/conda environment X, Python/conda environment Y, R, and Spark).  \n",
    "- **nb_conda_kernels** In a Jupyter Notebook, you can choose the kernel you want to run, but if you're working in Python then normally your choice is limited to choosing a version of Python.  This wonderful little package adds whatever conda environments you have set up to the list of options.  That makes it much easier to be certain about which version of any package you are running.\n",
    "- **nbdev** provides machinery for turning a Jupyter Notebook directly into runnable Python code.  It was developed by the _fastai_ team, but has broad utility.  See https://github.com/fastai/nbdev.  Note that it requires a couple of additional steps in Github (basically, you create a repository from a special template), and a significant mindshift to accomodate its workflow.\n",
    "\n",
    "**Warning about conda channels**: I have occasionally caused myself huge headaches by combining different package repository channels (for example, by installing a program with the main conda channel and then using conda-forge to add some package to my new program's environment).  Some very experienced conda wizards have told me that it's best to stick to one channel when possible.  Mixing conda and pip may also cause problems eventually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detectron2\n",
    "Detectron2 is difficult to install and there are distinct versions that are tightly linked to particular CUDA versions (see the table https://detectron2.readthedocs.io/en/latest/tutorials/install.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting work\n",
    "```bash\n",
    "#My normal start-up sequence\n",
    "ssh <user@12.34.567.89>\n",
    "conda activate <env>\n",
    "\n",
    "#notebook-dir will be ignored if you have spaces on either side of the '='\n",
    "jupyterlab --notebook-dir=~/<project_root> --port=8888 #or other port\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}