Use environment.yml to create env

dkirkby · Jan 2, 2019 · 19709d2 · 19709d2
1 parent 44debf4
commit 19709d2
Show file tree

Hide file tree

Showing 2 changed files with 127 additions and 52 deletions.
diff --git a/environment.yml b/environment.yml
@@ -0,0 +1,27 @@
+## Conda environment for a UC Irvine course on Machine Learning & Statistics.
+## Details at:
+## https://nbviewer.jupyter.org/github/dkirkby/MachineLearningStatistics/blob/master/notebooks/Setup.ipynb
+
+name: MLS
+channels:
+  - anaconda
+  - conda-forge   # needed for jupyterlab, jupyter_contrib_nbextensions, emcee
+dependencies:
+  - python=3.6    # tensorflow does not support 3.7 yet
+  - pip           # ensures that any pip installs only modify this env
+  - ipython
+  - jupyter
+  - jupyterlab    # needed for "jupyter lab"
+  - ipykernel     # ensures that (base) nb_conda creates a kernel for this env
+  - numpy
+  - scipy
+  - pandas
+  - matplotlib
+  - seaborn
+  - scikit-learn
+  - hdf5
+  - pytables
+  - pillow
+  - libiconv       # needed by jupyter_contrib_nbextensions on some platforms
+  - jupyter_contrib_nbextensions
+  - emcee
diff --git a/notebooks/Setup.ipynb b/notebooks/Setup.ipynb
@@ -15,7 +15,9 @@
     "\n",
     "Content is maintained on [github](github.com/dkirkby/MachineLearningStatistics) and distributed under a [BSD3 license](https://opensource.org/licenses/BSD-3-Clause).\n",
     "\n",
-    "[Table of contents](Contents.ipynb)"
+    "[Table of contents](Contents.ipynb)\n",
+    "\n",
+    "> Comments intended for more advanced users are indented below like this."
    ]
   },
   {
@@ -30,16 +32,49 @@
    "metadata": {},
    "source": [
     "- [Create a github account](https://github.com/join) if you don't already have one.\n",
-    "- [Install the git command-line tools](https://git-scm.com/downloads) on your computer, if necessary.\n",
-    "- Install the python 3.6 version of [anaconda](https://www.anaconda.com/download/) on your computer, if necessary.\n",
+    "- [Download](https://git-scm.com/downloads) and install the git command-line tools on your computer, if necessary.\n",
+    "- [Download](https://www.anaconda.com/download/) and install the current python 3.x version of the [anaconda distribution](https://www.anaconda.com/distribution/) on your computer, if necessary.\n",
+    "\n",
+    "> If you already have an older version of anaconda installed that you want to keep, the instructions below create a new environment that should not disrupt your previous work and you can probably continue with the instructions below after running:\n",
+    "```\n",
+    "conda update conda\n",
+    "```\n",
     "\n",
-    "This course assumes a basic familiarity with the core python language. If you are rusty or still learning, I recommend the free ebook [A Whirlwind Tour of Python](http://www.oreilly.com/programming/free/a-whirlwind-tour-of-python.csp), which is *\"a fast-paced introduction to essential components of the Python language for researchers and developers who are already familiar with programming in another language\"*.\n",
+    "This course assumes a basic familiarity with the core python language. If you are rusty or still learning, I recommend the free ebook [A Whirlwind Tour of Python](https://www.oreilly.com/programming/free/files/a-whirlwind-tour-of-python.pdf), which is *\"a fast-paced introduction to essential components of the Python language for researchers and developers who are already familiar with programming in another language\"*.\n",
     "\n",
-    "If you are currently using python 2.x and reluctant to move to python 3, read [this](https://wiki.python.org/moin/Python2orPython3) and [this](http://www.python3statement.org/).\n",
+    "> If you are currently using python 2.x and reluctant to move to python 3, read [this](https://wiki.python.org/moin/Python2orPython3) and [this](http://www.python3statement.org/).\n",
     "\n",
     "No previous experience with git or github is necessary for this course (but they are useful research tools so worth learning - [here](https://guides.github.com/introduction/git-handbook/) is a good starting point).  If you are finding the git learning curve to be steep, you are [not alone](https://explainxkcd.com/wiki/index.php/1597:_Git)."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Install course material"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Clone the course material from github with the following command, which will create a subdirectory called :\n",
+    "```\n",
+    "git clone https://github.com/dkirkby/MachineLearningStatistics.git\n",
+    "```\n",
+    "\n",
+    "> This command may prompt you for your github username and password but you can streamline future [github access using ssh](https://help.github.com/articles/which-remote-url-should-i-use/).\n",
+    "\n",
+    "You should now have a subdirectory called `MachineLearningStatistics`.  Enter it for the remaining steps:\n",
+    "```\n",
+    "cd MachineLearningStatistics\n",
+    "```\n",
+    "Any instructions below that assume you are in this subdirectory will be prefaced with the shell comment:\n",
+    "```\n",
+    "# cd MachineLearningStatistics/\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -51,28 +86,57 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will use the [conda command](https://conda.io/docs/commands.html) to create a standard [python environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for this course. These instructions assume that you have already satisfied the prerequisites."
+    "We will use the [conda command](https://conda.io/docs/commands.html) to create a standard [python environment](https://conda.io/docs/user-guide/tasks/manage-environments.html) for this course."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Create a new environment by entering (or pasting) the following command at a shell prompt. **Enter the command on one line**, even though it appears on two lines below:\n",
+    "Create a new environment using the following command at a shell prompt:\n",
     "```\n",
-    "conda create -n MLS python=3.6 pip ipython jupyter numpy scipy pandas\n",
-    "  matplotlib seaborn scikit-learn hdf5 pytables pillow\n",
+    "# cd MachineLearningStatistics/\n",
+    "conda env create -f environment.yml\n",
     "```\n",
+    "This command may run for several minutes.\n",
+    "\n",
+    "> We are using python version 3.6 for this environment although the anaconda default (as of Jan 2019) is 3.7, since the tensorflow package is [not yet compatible](https://github.com/tensorflow/tensorflow/issues/20444) with 3.7.\n",
+    "\n",
     "Activate the new environment using (this should add \"(MLS)\" to your command prompt, as a reminder of your current environment):\n",
     "```\n",
-    "source activate MLS\n",
+    "conda activate MLS\n",
+    "```\n",
+    "\n",
+    "> To \"deactivate\" this enironment and return to your default base environment, use:\n",
+    "```\n",
+    "conda deactivate\n",
+    "```\n",
+    "\n",
+    "> Older versions of conda used a [different syntax](https://conda.io/docs/user-guide/tasks/manage-environments.html#activating-an-environment) to activate and deactivate an environment.\n",
+    "\n",
+    "Install the [tensorflow](https://www.tensorflow.org/) machine-learning framework:\n",
+    "```\n",
+    "pip install tensorflow\n",
+    "```\n",
+    "\n",
+    "> This pip command will only install tensorflow into your MLS environment (since it uses a version of pip that is local to the MLS environment).\n",
+    "\n",
+    "> If you are installing onto a linux or windows system with a [CUDA-enabled GPU card](https://www.tensorflow.org/install/gpu), use instead:\n",
+    "```\n",
+    "pip install tensorflow-gpu\n",
+    "```\n",
+    "\n",
+    "Install the [pytorch](https://pytorch.org/) machine-learning framework. The exact command is different for Mac and linux/windows. On a Mac, use:\n",
+    "```\n",
+    "conda install pytorch torchvision -c pytorch\n",
     "```\n",
-    "Add some additional packages from other sources (details [here](https://github.com/ipython-contrib/jupyter_contrib_nbextensions#conda) and [here](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/issues/1153)):\n",
+    "On a windows or linux system, use:\n",
     "```\n",
-    "conda install -c conda-forge libiconv jupyter_contrib_nbextensions\n",
-    "conda install -c astropy emcee astroml\n",
-    "pip install wpca autograd tensorflow edward\n",
+    "conda install pytorch-cpu torchvision-cpu -c pytorch\n",
     "```\n",
+    "\n",
+    "> See [here](https://pytorch.org/get-started/locally/) for alternate install commands for a  linux or windows system with a CUDA-enabled GPU card.\n",
+    "\n",
     "Enable a jupyter notebook [extension](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/exercise2) we will use for in-class exercises:\n",
     "```\n",
     "jupyter nbextension enable exercise2/main\n",
@@ -83,37 +147,20 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In case something goes wrong with your installation and you want to start again, use:\n",
+    "> In case something goes wrong with your installation and you want to start again, shutdown any jupyter sessions with the old environment, then use:\n",
     "```\n",
+    "conda deactivate\n",
     "conda remove --name MLS --all\n",
-    "```\n",
-    "You will need to shutdown any jupyter sessions with the old environment first."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Install course material"
+    "```"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Clone the course material from github with the following command, which will create a subdirectory called `MachineLearningStatistics`:\n",
+    "Finally, install the course code and data into your new environment using:\n",
     "```\n",
-    "git clone https://github.com/dkirkby/MachineLearningStatistics.git\n",
-    "```\n",
-    "This should ask you for your github username and password (but you can streamline future [github access using ssh](https://help.github.com/articles/which-remote-url-should-i-use/)).\n",
-    "\n",
-    "Activate the course environment, if necessary (check your command prompt, but it doesn't do any harm to reactivate the current environment):\n",
-    "```\n",
-    "source activate MLS\n",
-    "```\n",
-    "Install the course code and data using:\n",
-    "```\n",
-    "cd MachineLearningStatistics\n",
+    "# cd MachineLearningStatistics/\n",
     "pip install .\n",
     "```"
    ]
@@ -131,18 +178,17 @@
    "source": [
     "To launch the [notebook server](http://jupyter-notebook.readthedocs.io/en/stable/notebook.html) at any time, you can now use:\n",
     "```\n",
-    "[[MachineLearningStatistics]]\n",
-    "source activate MLS\n",
+    "# cd MachineLearningStatistics/\n",
+    "conda activate MLS\n",
     "cd notebooks\n",
-    "jupyter notebook\n",
+    "jupyter lab\n",
     "```\n",
-    "Note that `[[MachineLearningStatistics]]` is a reminder that you must be in your `MachineLearningStatistics` directory before typing the following commands.  If you are unsure about this, refer to the [pwd](http://www.linfo.org/pwd.html) and [cd](http://www.linfo.org/cd.html) commands.\n",
     "\n",
-    "**Windows users:** Wherever you see `source activate MLS`, use `activate MLS` instead.  Details [here](https://conda.io/docs/user-guide/tasks/manage-environments.html#activating-an-environment).\n",
+    "> These instructions launch [JupyterLab](https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906) rather than the older notebook server.\n",
     "\n",
     "Click on `Contents.ipynb` if this is your first time doing this, to check that you can view a notebook.\n",
     "\n",
-    "*(For git experts: you will normally be working on the master branch to simplify the workflow. This means that your local work must be discarded or saved to another branch each time you update, using the instructions below).*"
+    "These instructions allow you to modify and run python code on your local computer from within your browser. If you just want to view these notebooks online, try this [nbviewer link](https://nbviewer.jupyter.org/github/dkirkby/MachineLearningStatistics/tree/master/notebooks/)."
    ]
   },
   {
@@ -158,41 +204,43 @@
    "source": [
     "You can skip this section if you are installing for the first time, but remember these instructions for later.\n",
     "\n",
+    "> For git experts: you will normally be working on the master branch to simplify the workflow. This means that your local work must be discarded or saved to another branch each time you update, using the instructions below.\n",
+    "\n",
     "The first step is to \"factory reset\" your installation before getting the updates. The simplest method is to throw away any changes you have made using:\n",
     "```\n",
-    "[[MachineLearningStatistics]]\n",
+    "# cd MachineLearningStatistics/\n",
     "git checkout master\n",
     "git reset --hard\n",
     "```\n",
-    "Alternatively, you can keep a permanent record of your changes in a [git branch](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) with a name of your choice, for example \"08-Jan-2018\":\n",
+    "Alternatively, you can keep a permanent record of your changes in a [git branch](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) with a name of your choice, for example \"08-Jan-2019\":\n",
     "```\n",
-    "[[MachineLearningStatistics]]\n",
-    "git checkout -b \"08-Jan-2018\"\n",
+    "# cd MachineLearningStatistics/\n",
+    "git checkout -b \"08-Jan-2019\"\n",
     "git commit -a -m \"Save work in progress\"\n",
     "git checkout master\n",
     "```\n",
     "\n",
     "The second step is to download the changes from github:\n",
     "```\n",
-    "[[MachineLearningStatistics]]\n",
+    "# cd MachineLearningStatistics/\n",
     "git pull\n",
     "```\n",
     "If this commands reports `Already up-to-date.` then there are no updates to download.\n",
     "\n",
     "The final step is to update your local python environment:\n",
     "```\n",
-    "[[MachineLearningStatistics]]\n",
-    "source activate MLS\n",
+    "# cd MachineLearningStatistics/\n",
+    "conda activate MLS\n",
     "pip install . --upgrade\n",
     "```"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python [conda env:MLS]",
    "language": "python",
-   "name": "python3"
+   "name": "conda-env-MLS-py"
   },
   "language_info": {
    "codemirror_mode": {
@@ -204,7 +252,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.4"
+   "version": "3.6.7"
   }
  },
  "nbformat": 4,