Merge branch '1.0.x' into 'master'

CHANGELOG.md

@@ -11,6 +11,22 @@ Features:
 * Configuration file contains the walltime taken by each run
 * It is now possible to upload or download any file via its full path
 
+1.0.11 (2017-11-05)
+-------------------
+
+Bugfixes:
+* Write timestamp in config with timezone offset
+* Always fix up PATH on MacOS (to pick up Docker)
+* Don't have reprounzip-qt choke on MacOS's `-psn_x_xxxx` arguments
+* Fix "Couldn't find reprounzip command" in the GUI on Windows if installed from the installer and opened a package by double-click
+
+Enhancements:
+* Improve DOT graph layout
+* Don't silently overwrite output with `reprounzip graph`
+* Add `--regex-include` to `reprounzip graph`
+* `reprozip trace` now returns 125 for tracing errors (previously 1), and returns the traced program's exit code otherwise (previously always 0)
+* Manpages are available
+
 1.0.10 (2017-07-10)
 -------------------
 

README.md

@@ -4,7 +4,7 @@
 [![Documentation Status](https://readthedocs.org/projects/reprozip/badge/?version=latest)](https://docs.reprozip.org/en/latest/)
 [![Matrix](https://img.shields.io/badge/chat-matrix.org-blue.svg)](https://riot.im/app/#/room/#reprozip:matrix.org)
 [![status](https://img.shields.io/badge/JOSS-10.21105%2Fjoss.00107-green.svg)](http://joss.theoj.org/papers/b578b171263c73f64dfb9d040ca80fe0)
-[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.825149-green.svg)](https://doi.org/10.5281/zenodo.825149)
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.1042123-green.svg)](https://doi.org/10.5281/zenodo.1042123)
 
 ReproZip
 ========

docs/conf.py

@@ -53,6 +53,13 @@
 project = u'ReproZip'
 copyright = u'2014-2017, New York University'
 
+authors = [u'Fernando Chirigati', u'Remi Rampin',
+           u'Juliana Freire', u'Dennis Shasha']
+if len(authors) <= 1:
+    authors_str = u', '.join(authors)
+else:
+    authors_str = u', '.join(authors[:-1]) + u', and ' + authors[-1]
+
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
@@ -146,8 +153,8 @@
 # html_logo = None
 
 # The name of an image file (relative to this directory) to use as a favicon of
-# the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
+# the docs.  This file should be a Windows icon file (.ico) being 16x16 or
+# 32x32 pixels large.
 #
 # html_favicon = None
 
@@ -261,8 +268,7 @@
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     (master_doc, 'ReproZip.tex', u'ReproZip Documentation',
-     u'Fernando Chirigati, Remi Rampin, Juliana Freire, and Dennis Shasha',
-     'manual'),
+     authors_str, 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -296,10 +302,7 @@
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'reprozip', u'ReproZip Documentation',
-     [u'Fernando Chirigati', u'Remi Rampin', u'Juliana Freire', u'Dennis Shasha'], 1)
-]
+man_pages = []
 
 # If true, show URL addresses after external links.
 #
@@ -313,9 +316,9 @@
 #  dir menu entry, description, category)
 texinfo_documents = [
     (master_doc, 'ReproZip', u'ReproZip Documentation',
-     u'Fernando Chirigati, Remi Rampin, Juliana Freire, and Dennis Shasha',
-     'ReproZip',
-     'Allows the reproducibility of command-line experiments in a different environment',
+     authors_str, 'ReproZip',
+     u"Allows the reproducibility of command-line experiments in a different "
+     "environment",
      'Miscellaneous'),
 ]
 

docs/developerguide.rst

@@ -10,7 +10,7 @@ Development happens on `GitHub <https://github.com/ViDA-NYU/reprozip>`__; bug re
 
 Continuous testing is provided by `Travis CI <https://travis-ci.org/ViDA-NYU/reprozip>`__. Note that ReproZip supports both Python 2 and 3. Test coverage is not very high because there are a lot of operations that are difficult to cover on Travis (for instance, Vagrant VMs cannot be used over there).
 
-If you have any questions or need help with the development of an unpacker or plugin, please use our development mailing-list at `reprozip-dev@vgc.poly.edu <https://vgc.poly.edu/mailman/listinfo/reprozip-users>`__.
+If you have any questions or need help with the development of an unpacker or plugin, please use our development mailing-list at `dev@reprozip.org <https://vgc.poly.edu/mailman/listinfo/reprozip-users>`__.
 
 Writing Unpackers
 -----------------
@@ -89,4 +89,4 @@ Since version 0.4.1, `reprounzip` has signals that can be used to hook in plugin
 Final Observations
 ------------------
 
-After reading this page, reading the source code of one of the "official" unpackers is probably the best way of understanding how to write your own. They should be short enough to be easy to grasp. Should you have additional questions, do not hesitate to use our development mailing-list: `reprozip-dev@vgc.poly.edu`.
+After reading this page, reading the source code of one of the "official" unpackers is probably the best way of understanding how to write your own. They should be short enough to be easy to grasp. Should you have additional questions, do not hesitate to use our development mailing-list: `dev@reprozip.org`.

docs/gui.rst

@@ -8,33 +8,35 @@ ReproUnzip GUI
 Installation
 ============
 
+..  warning:: ``reprounzip-qt`` is Python 2 only. You can't install it with Python 3 or Anaconda3 (without a virtual environment)!
+
 *reprounzip-qt* comes with the installer on `Windows <http://reprozip-files.s3-website-us-east-1.amazonaws.com/windows-installer>`_ and `Mac <http://reprozip-files.s3-website-us-east-1.amazonaws.com/mac-installer>`_. If you used one of these, you will be able to double click on any ``.rpz`` file to boot up the GUI.
 
-If you are using Anaconda, you can install *reprounzip-qt* from anaconda.org::
+If you are using Anaconda2, you can install *reprounzip-qt* from anaconda.org::
 
     $ conda install --channel vida-nyu reprounzip-qt
 
 Otherwise, you will need to `install PyQt4 <https://www.riverbankcomputing.com/software/pyqt/download>`__ before you can install *reprounzip-qt* from pip (on Debian or Ubuntu, you can use ``apt-get install python-qt4``).
 
 On Linux, setting it as the default to open ``.rpz`` files is a bit more involved. Once the application is setup, run `this script <https://gist.github.com/remram44/0092c0b27269cfd0e5530428612d9309>`__ by  opening up a terminal window and entering ``bash register-linux.sh``. From there, you can back-click on a ``.rpz`` file, and set the default application to ReproUnzip. The following is an example of this process on Ubuntu 16.04:
 
-.. image:: figures/reprounzip-qt-defaultApp.png
+..  image:: figures/reprounzip-qt-defaultApp.png
 
 If ReproUnzip doesn't appear right away, simply click "Other Application" and find it in the menu, alphabetically.
 
-.. image:: figures/reprounzip-qt-defaultApp-1.png
+..  image:: figures/reprounzip-qt-defaultApp-1.png
 
 Usage
 ============
 
 The first tab in the window that appears is for you to set up the experiment. This will allow you to choose which `unpacker <unpacking.html#unpackers>`_ you'd like to use to reproduce the experiment, and in which directory you'd like to unpack it.
 
-.. image:: figures/reprounzip-qt.png
+..  image:: figures/reprounzip-qt.png
 
 After successfully unpacking, you'll be prompted to run the experiment in the second tab. You can choose which run you want to execute, though the default is to have all runs selected. ReproUnzip will detect the order of the runs and reproduce the experiment accordingly.
 
-.. image:: figures/reprounzip-qt-1.png
+..  image:: figures/reprounzip-qt-1.png
 
 Clicking "Manage Files" will bring up options to download the input and output data of the original experiment, and upload your own data to use it in the same experiment. You'll notice input files are marked ``[I]`` and output files are marked ``[O]``. ``[IO]`` are both input and output files.
 
-.. image:: figures/reprounzip-qt-manageFiles.png
+..  image:: figures/reprounzip-qt-manageFiles.png

docs/index.rst

@@ -23,8 +23,9 @@ Contents
     packing
     unpacking
     graph
-    vistrails
+    jupyter
     gui
+    vistrails
     faq
     troubleshooting
     unpacked-format
@@ -38,6 +39,6 @@ Links
 * `GitHub repository <https://github.com/ViDA-NYU/reprozip>`__
 * Mailing lists:
 
-  * `reprozip-users@vgc.poly.edu <https://vgc.poly.edu/mailman/listinfo/reprozip-users>`__ (users)
+  * `users@reprozip.org <https://vgc.poly.edu/mailman/listinfo/reprozip-users>`__ (users)
 
-  * `reprozip-dev@vgc.poly.edu <https://vgc.poly.edu/mailman/listinfo/reprozip-dev>`__ (developers)
+  * `dev@reprozip.org <https://vgc.poly.edu/mailman/listinfo/reprozip-dev>`__ (developers)

docs/jupyter.rst

@@ -0,0 +1,72 @@
+..  _reprozip-jupyter:
+
+Making Jupyter Notebooks Reproducible with ReproZip
+***************************************************
+
+**reprozip-jupyter** is a plugin for `Jupyter Notebooks <https://jupyter.org>`__, a popular open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. These are valuable documents for data cleaning, analysis, writing executable papers/articles, and more. However, Jupyter Notebooks are subject to dependency hell like any other application -- just the Notebook is not enough for full reproducibility. We have written a ReproZip plugin for Jupyter Notebooks to help users automatically capture dependencies (including data, environment variables, etc.) of Notebooks and also automatically set up those dependencies in another computing environment.
+
+Installation
+============
+You can install *reprozip-jupyter* with pip::
+
+	  $ pip install reprozip-jupyter
+
+Or Anaconda::
+
+		$ conda install reprozip-jupyter
+
+Once successfully installed, you should then enable the plugin for both the client and server side of Jupyter Notebooks::
+
+		$ jupyter nbextension install --py reprozip_jupyter --user
+		$ jupyter nbextension enable --py reprozip_jupyter --user
+		$ jupyter serverextension enable --py reprozip_jupyter --user
+
+Once these steps are completed, when you start a Jupyter Notebook server, you should be able to see the ReproZip button in your notebook's toolbar.
+
+Packing
+=======
+
+Once you have a notebook that executes the way you want, you can trace and pack all the dependencies, data, and provenance with *reprozip-jupyter* by simply clicking the button on the notebook's toolbar:
+
+..  image:: figures/rzj-button.png
+
+The notebook will execute from top-to-bottom and *reprozip-jupyter* traces that execution. If there are no errors in the execution, you'll see two pop-ups like this one after the other:
+
+..  image:: figures/rzj-running.png
+
+*reprozip-jupyter* will name the resulting ReproZip package (*.rpz*) as ``notebookname_datetime.rpz`` and save it to the same working directory the notebook is in:
+
+..  image:: figures/rzj-pkg.png
+
+Note that the notebook file itself (``.ipynb``) is not included in the package, so you should share or archive both of those files. The reason is that a lot of services can render notebooks (GitHub, OSF...), and they wouldn't be able to if it was in the RPZ file.
+
+Unpacking
+=========
+
+Now, anyone can rerun the Jupyter notebook, with all dependencies automatically configured. First, they would need to install *reprounzip* and the *reprounzip-docker* plugin (see the :ref:`installation steps <install>`). Second, they need to download or otherwise acquire the ``.rpz`` file and original ``.ipynb`` notebook they'd like to reproduce.
+
+To reproduce the notebook using the GUI, follow these steps:
+
+1. Double-click the .rpz file.
+2. The first tab in the window that appears is for you to set up how you'd like ReproUnzip to unpack and configure the contents of the *.rpz*. Choose docker as your unpacker, and choose the directory you'd like to unpack into.
+3. Make sure the Jupyter Integration is checked, and click Run experiment:
+
+..  image:: figures/rzj-setup.png
+
+4. This second table allows you to interact with and rerun the notebook. All you need to do is click 'Run Experiment' and the Jupyter Notebook home file list should pop up in your default browser (if not, navigate to ``localhost:8888``). Open the notebook, and rerun with every dependency configured for you!
+
+..  image:: figures/rzj-run.png
+
+On the command line, you would:
+
+1. Set up the experiment using *reprounzip-docker*::
+
+		$ reprounzip docker setup <package.rpz> <directory>
+
+2. Rerun the notebook using *reprozip-jupyter*::
+
+		$ reprozip-jupyter run <directory>
+
+3. The Jupyter Notebook home file list should pop up in your default browser (if not, navigate to ``localhost:8888``).
+4. Open the notebook, and rerun with every dependency configured for you!
+

docs/man/reprounzip-docker.1

@@ -0,0 +1,85 @@
+.\" Manpage for reprounzip
+.\" Contact reprozip-dev@vgc.poly.edu to correct errors or typos.
+.TH man 1 "4 November 2017" "1.0.10" "reprounzip\-docker man page"
+.SH NAME
+reprounzip\-docker \- Docker unpacker for reprounzip
+.SH SYNOPSIS
+reprounzip [\-v] docker [\-h] [\-\-version] [\-\-docker\-cmd DOCKER] ...
+.SH DESCRIPTION
+reprounzip\-docker is the unpacker plugin for reprounzip that allows reproducing packed experiments on different environments by using a Docker container. You will need Docker to be installed on your machine to use it.
+.SH OPTIONS
+.SS General options:
+.TP
+.B \-h, \-\-help
+show this help message and exit
+.TP
+.B \-\-version
+show program's version number and exit
+.TP
+.B \-v, \-\-verbose
+augments verbosity level
+.TP
+.B \-\-docker\-cmd DOCKER
+changes the location or command used to run the Docker client. This option is split on spaces, for example you can use
+\-\-docker\-cmd 'sudo /opt/bin/docker'
+
+.SS Commands:
+.TP
+.BI setup " [\-\-base\-image IMAGE] [\-\-distribution DISTRIBUTION] [\-\-install\-pkgs] [\-\-image\-name NAME] [\-\-docker\-option OPTION] <package.rpz> <directory>"
+Unpacks an experiment and sets it up for reproduction in the specified directory. A Docker image
+.I NAME
+will be created from the specified base
+.I IMAGE
+by adding the experiment's content on top of it. You can use \-\-docker\-option to pass additional options to docker build for example \-\-docker\-option=\-\-squash.
+.TP
+.BI run " [\-\-expose\-port PORT] [\-\-enable\-x11] [\-\-x11\-display DISPLAY] [\-\-tunneled\-x11] [\-d] [\-\-docker\-option OPTION] [\-\-pass\-env VAR] [\-\-set\-env VAR[=val]] <directory> [run] [\-\-cmdline ...]"
+Runs an experiment that has been unpacked to a directory.
+.TP
+.BI upload " <directory> [localfile:experimentfile] [...]"
+Replaces a file inside the experiment image with your own. This allows you to reproduce an experiment in its original environment, but with modified data or configuration. Running this command with only the
+.I directory
+will list all the named files that you can replace with this command.
+.TP
+.BI download " <directory> [experimentfile[:localfile]] [...]"
+Download a file from the experiment container to your filesystem. If
+.I localfile
+is not specified, a file of the same name will be written in the current directory. Running this command with only the
+.I directory
+will list all the named files that you can download with this command.
+.TP
+.BI reset " <directory>"
+Reset the experiment to the state it was when it was first unpacked. This undoes the effects of
+.BR upload " and " run
+commands on the experiment's environment.
+.TP
+.BI destroy " <directory>"
+Removes an unpacked experiment directory, and the associated Docker images.
+.SH SEE ALSO
+.TP
+The ReproZip website
+https://www.reprozip.org/
+.TP
+ReproZip's GitHub repository
+https://github.com/ViDA\-NYU/reprozip
+.TP
+The Docker website
+https://www.docker.com/
+.SH BUGS
+Please report bugs on our mailing-list at users@reprozip.org or on GitHub at https://github.com/ViDA\-NYU/reprozip/issues.
+.SH AUTHOR
+.RB "ReproZip is being developed at" " New York University" .
+
+The team includes:
+.RS
+.nf
+Remi Rampin
+Fernando Chirigati
+Vicky Steeves
+Juliana Freire
+Dennis Shasha
+.fi
+.RE
+.SH COPYRIGHT
+Copyright 2014-2017 New York University.
+
+.RB "Licensed under a" " BSD 3-Clause license." " See the LICENSE file included with the software for details."