utils/Best Practices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Best Practices\n",
    "The goal of this document is to present some important tips about Reproducible Research. The description here shows what worked and what didn't work during my research in the course class IA369-Z from FEEC Unicamp. Here you will find *my lessons learned* during the course, and I hope they will help you with your research.\n",
    "\n",
    "## Reproducible Research\n",
    "You must make Reproducible Research during your research. It is a way to research, and it is not an end step, like: \"Ok, now I will put my files on Github and everybody will be able to reproduce my experiments\". Reproducible Research is a practical approach that will help you to improve the **quality** of your work, to **validate** your findings and to **do relevant contributions in your area**. When other researchers can reproduce your experiment, they can use it and create from your work. Even if they find problems in your experiment, surely they will let you know and that is a good thing.\n",
    "\n",
    "I think that sometimes we are so concentrated in the validation of our hypothesis, and in the writing (paper) of our insights that we forget to report detailed steps and decisions (for us obvious), making the reproduction impossible (for other researchers and even yourself).\n",
    "\n",
    "What helped me during the experiments was to keep in my mind *\"A human being will read my paper, code, and notes.\"*\n",
    "\n",
    "If you are still not convinced, please, read these references: [Useless Studies](http://brasil.elpais.com/brasil/2017/01/10/internacional/1484073680_523691.html?id_externo_rsoc=FB_CC),  [Manifesto from Nature](https://www.nature.com/articles/s41562-016-0021).\n",
    "\n",
    "\n",
    "## Resources for Reproducibility\n",
    "\n",
    "Now, I hope, you are motivated to do Reproducible Research. Then you can start by reading this [10 Simple Rules](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285). \n",
    "\n",
    "After discussions during the classes, we agreed upon the following elements of Reproducible Research:\n",
    "* Version Control\n",
    "* Workflow\n",
    "* Data repository\n",
    "* Code development\n",
    "* Documentation\n",
    "* Environment\n",
    "\n",
    "I will comment on each element. To be clear, I will separate the tips in DO (it works) and DON'T (it doesn't work):\n",
    "\n",
    "### Workflow\n",
    "\n",
    "**---- DO ----**\n",
    "* For me, to design the workflow must be the first step of your research. Even in the beginning, it will help you to think and to draw the flow of the research with the inputs, outputs, dependencies and data involved. Moreover, it will help to plan your research;\n",
    "* Save your editable files (for any tool), because certainly the workflow will change during the process;\n",
    "* [Draw.io](http://draw.io/) is a good tool and it is integrated with Git (the integration works very well);\n",
    "\n",
    "**---- DON'T ----**\n",
    "* Do not underestimate this step, even in the first version, insert as many details as possible;\n",
    "\n",
    "### Version Control\n",
    "\n",
    "**---- DO ----**\n",
    "* After planning your research, \"befriend\" the version control;\n",
    "* [Git](https://git-scm.com/) is a good and disseminated option (and you will find a lot of documentation);\n",
    "* Select a repository, [Github](https://github.com/) and [Bitbucket](https://bitbucket.org/) are famous ones;\n",
    "* Use the version control DURING your work, making ``commits`` in changes that you want to save;\n",
    "* If you are new in git, there are some good guides: [Try Git](https://try.github.io/) and [Github Guides](https://guides.github.com/) \n",
    "\n",
    "**---- DON'T ----**\n",
    "* Do not leave to the end of your research to put the codes (and related files) in the repository. Make your repository in the beginning.\n",
    "\n",
    "### Data Repository\n",
    "\n",
    "**---- DO ----**\n",
    "* This is an important step. Provide in your repository the raw and/or processed dataset;\n",
    "* Make sure you are clear, in the documentation, about the dataset;\n",
    "* If you are using dataset from another researcher, you must reference it.\n",
    "\n",
    "**---- DON'T ----**\n",
    "* Do not make manual manipulations in dataset. Use codes or scripts to do that, because, you can easily forget these changes.\n",
    "\n",
    "### Code Development and Literate Programming\n",
    "\n",
    "Literate Programming was a new concept for me, but much important. So, to do Reproducible Research you must understand it:\n",
    ">*“Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to humans what we want the computer to do.” Donald Knuth*\n",
    "\n",
    "** ---- DO ----**\n",
    "* Select a good environment to develop your codes and texts (Literate Programming);\n",
    "* [Jupyter Notebook](http://jupyter.org/) is a good option. They recommend to install [Anaconda](https://www.continuum.io/downloads) that comes with Python and Jupyter;\n",
    "* Learn about [Markdown](https://blog.da2k.com.br/2015/02/08/aprenda-markdown/), you will use this kind of text to explain your codes and experiments;\n",
    "* Code with [good practices](http://www.devmedia.com.br/boas-praticas-de-programacao/31163?utm_content=buffer1cddf&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer) of programming, that will make a big difference; \n",
    "* Report libraries installed to execute your codes (including versions);\n",
    "* In your files, put a cell with text explaining the decisions or steps, and after it, the cell(s) with codes. \n",
    "\n",
    "**---- DON'T ----**\n",
    "* When you install the Anaconda, make sure you set the right path. It looks like a foolish thing, but I installed thinking \"I will select any folder, because after I can change it\", but when I needed to define the paths to repository, I couldn't, and I had to desinstall and install again.\n",
    "\n",
    "### Documentation\n",
    "I considered as documentation my official paper, so thinking to be more productive, I used the Jupyter Notebook. The experience was good for me, because you can put in the same file cells with codes and texts. The problem was when I converted Markdown in [TEX (LaTeX)](https://www.latex-project.org/), it was not as productive as I had hoped. I had to correct many things in the file. Although it was worth it, at least in the first version.\n",
    "\n",
    "Keep it in your mind: \"The Notebook file will be converted in Markdown, and then in TEX\". Well, you can ask: why do we have to convert it in TEX? Because you will want to insert citation of references and figures, use templates and format your text. Markdown has limitations for these needs.\n",
    "\n",
    "My recommendation: If you will use in your paper many code cells, then use Jupyter Notebook and make the convertions **otherwise, write directly in LaTeX**. \n",
    "\n",
    "If in your case you have many code cells:\n",
    "\n",
    "**---- DO ----**\n",
    "* Create your [bib tex](http://www.bibtex.org/) file with your references; \n",
    "* Write your paper (in Jupyter Notebook) with the LaTeX command of reference to figures and tables `\\ref{}` and citations `\\cite{}`;\n",
    "* Convert your file:\n",
    "\n",
    "```\n",
    "jupyter nbconvert --to markdown <jupyter.ipynb> \n",
    "```\n",
    "\n",
    "```\n",
    "pandoc -f markdown -t latex <filename>.md -o filename.tex\n",
    "```\n",
    "\n",
    "* You can use [Overleaf](https://www.overleaf.com/). It is an online tool to create, edit and share LaTeX files in a collaborative way (and I recommend it); \n",
    "* Open your generated tex file and make the necessary corrections (figures, lists and so on);\n",
    "\n",
    "**---- DON'T ----**\n",
    "* Do not leave this step to the end of your research, since the documentation needs to be build gradually;\n",
    "* There are some resources that make sense to be in the paper and others that do not. Do not insert codes that do not make sense in paper. For example, a call to code file, or codes which do not have a readable output (table, figure, etc.);\n",
    "* There is this reference to [Update Overleaf from a Jupyter Notebook](https://medium.com/thoughts-philosophy-writing/how-to-update-overleaf-from-a-juypter-notebook-5469b1405fdc), I used it, but it was not productive... I copied files manually many times; \n",
    "\n",
    "### Environment\n",
    "Based on my experience (including the course classes), the environment is the main limitation to reproduce computational research. We know that softwares have releases and dependencies and these can generate problems (e.g. compatibility). The options for environments are:\n",
    "\n",
    "1. Local - It is not a safe option. You have to report ALL dependencies and versions (including save as backups of installers);\n",
    "2. Virtual Machine - It is a good option, but you have the size problem. Whoever reproduces your research, he/she will need to have access to good Internet connection;\n",
    "3. [Docker](http://docker.com/) - It is the best option! It uses the concept of containers (which I recommend strongly).\n",
    "\n",
    "I used Local and Docker environments:\n",
    "\n",
    "**---- DO ----**\n",
    "* I recommend you learn about Docker (Some references: [Docker Basics](https://gitlab.com/daitan-learn/docker-basics), [Docker Lab Lessons](http://labs.play-with-docker.com/),[Blog](http://www.diego-garcia.info/2015/02/15/docker-por-onde-comecar/), [Mundo Docker](http://www.mundodocker.com.br/tag/docker-no-windows/), [Tutorials](https://www.digitalocean.com/community/tutorials/como-instalar-e-utilizar-o-docker-primeiros-passos-pt), [Blog ](https://woliveiras.com.br/posts/Criando-uma-imagem-Docker-personalizada/))\n",
    "* Important to know: **Image** is a configurated environment and **Container** is an instance of image running in your machine;\n",
    "* Verify if the applications you need in your project are already in [Docker Hub](https://hub.docker.com/). You can find images ready to use with the complete environment;\n",
    "* Make all your experiments (if possible) using containers. In others words, configurate your development environment using image instead of local;\n",
    "* You can use different images for each application you need (e.g. database, programming environment) and run them in parallel;\n",
    "* When you run an image, verify the correct parameters in `run` command (ports, volume mapping);\n",
    "* The command `docker run <image> -it` means you want interactivity and link with the container's shell. On the other hand, the command `docker run <image> -d`, means you want to run the image in background (take care, for someone who is not familiar with Docker, the terminal will not explicitly warn that the container is up).\n",
    "\n",
    "\n",
    "**---- DON'T ----**\n",
    "* Don't put the research data in the Docker image. Probably the data is the most important element of your research, and a container is not persistent.\n",
    "* Don't forget to stop and to remove the containers after the use:\n",
    "\n",
    "*Verify the containers*:\n",
    "```\n",
    "docker ps -a\n",
    "```\n",
    "\n",
    "*Stop all containers:*\n",
    "```\n",
    "docker stop $(docker ps -a -q)\n",
    "```\n",
    "\n",
    "*Remove all containers:*\n",
    "```\n",
    "docker rm $(docker ps -a -q)\n",
    "\n",
    "``` \n",
    "\n",
    "## Final Recommendations\n",
    "**---- DO ----**\n",
    "* As a final step of your research, when your results and paper are done, create the README files. The instructions in these files are important, so make sure you are clear about every step of your research. Ask some colleague to follow your instructions and to try to reproduce your experiment (another person will identify confusing explanations).\n",
    "\n",
    "**---- DON'T ----**\n",
    "* Don't put many README files, I did that, and my instructions were confusing."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}