Skip to content

Commit

Permalink
Pass over docs.
Browse files Browse the repository at this point in the history
  • Loading branch information
fchirigati committed May 6, 2015
1 parent 9d14cba commit 1bfa80a
Show file tree
Hide file tree
Showing 3 changed files with 53 additions and 50 deletions.
38 changes: 20 additions & 18 deletions docs/source/developerguide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,38 @@ Developer's Guide
General Development Information
-------------------------------

Development happens on `GitHub <https://github.com/ViDA-NYU/reprozip>`__; bug reports or feature requests are welcome. If you are interested in giving a hand, please do not hesitate to submit a pull request there.
Development happens on `GitHub <https://github.com/ViDA-NYU/reprozip>`__; bug reports and feature requests are welcome. If you are interested in giving us a hand, please do not hesitate to submit a pull request there.

Continuous testing is provided by `Travis CI <https://travis-ci.org/ViDA-NYU/reprozip>`__. Note that ReproZip supports both Python 2 and 3. Test coverage is not very high due to a lot of operations that are difficult to cover on Travis (Vagrant VMs and Docker containers cannot be used over there).

If you have any questions or need help with the development of an unpacker or plugin, please do not hesitate to use the development mailing-list at `reprozip-dev@vgc.poly.edu`.
If you have any questions or need help with the development of an unpacker or plugin, please use our development mailing-list at `reprozip-dev@vgc.poly.edu`.

Writing Unpackers
-----------------

ReproZip is divided into two steps. Packing gives a generic package containing the trace SQLite database, YAML configuration file (listing the paths, packages, and metadata such as command-line, environment variables, and input/output files), and actual files. In the second step, a package can be turned into a runnable form by reprounzip. This decoupling allows the reproducer to select the unpacker of his/her desire, and also means that when a new unpacker is released, users will be able to use it on their old packages.
ReproZip is divided into two steps. Packing gives a generic package containing the trace SQLite database, the YAML configuration file (listing the paths, packages, and metadata such as command line, environment variables, and input/output files), and actual files. In the second step, a package can be turned into a runnable form by *reprounzip*. This decoupling allows the reproducer to select the unpacker of his/her desire, and also means that when a new unpacker is released, users will be able to use it on their old packages.

The ViDA group maintains different unpackers: the two defaults ones (``directory`` and ``chroot``), ``vagrant`` (distributed as `reprounzip-vagrant <https://pypi.python.org/pypi/reprounzip-vagrant>`__) and ``docker`` (distributed as `reprounzip-docker <https://pypi.python.org/pypi/reprounzip-docker>`__). However, the interface is such that new unpackers can be easily added. While taking a look at the "official" unpackers' source is probably a good idea, this page gives some useful information about how they work.
The ViDA group maintains different unpackers: the defaults ones (``directory`` and ``chroot``), ``vagrant`` (distributed as `reprounzip-vagrant <https://pypi.python.org/pypi/reprounzip-vagrant>`__) and ``docker`` (distributed as `reprounzip-docker <https://pypi.python.org/pypi/reprounzip-docker>`__). However, the interface is such that new unpackers can be easily added. While taking a look at the "official" unpackers' source is probably a good idea, this page gives some useful information about how they work.

ReproZip pack format (``.rpz``)
ReproZip Pack Format (``.rpz``)
'''''''''''''''''''''''''''''''

An ``.rpz`` file is a tar.gz archive. It contains two directories: ``METADATA``, which contains meta-information from reprozip, and ``DATA``, which contains the actual files that were packed and that will be unpacked to the target directory for the experiment to use.
An ``.rpz`` file is a ``tar.gz`` archive that contains two directories: ``METADATA``, which contains meta-information from *reprozip*, and ``DATA``, which contains the actual files that were packed and that will be unpacked to the target directory for reproducing the experiment.

The ``METADATA/config.yml`` file is in the same format as the configuration file generated by tracing, that governs the packing step, with the difference that the ``additional_patterns`` section is gone (it has been expanded to the actual list of files during packing).
The ``METADATA/config.yml`` file is in the same format as the configuration file generated by *reprozip*, but without the ``additional_patterns`` section (at this point, it has already been expanded to the actual list of files while packing).

The ``METADATA/trace.sqlite3`` is the original trace generated by the C tracer, that is kept unmodified; it contains all the information about the experiment, in case the configuration file seems insufficient in some aspect. It is used for instance by the ``graph`` unpacker, so that it can tell the exact hierarchy of processes, with the executable images they execute, and the files they access (with the time and mode of these accesses). See also the :ref:`database schema for trace.sqlite3 <trace-schema>`.
The ``METADATA/trace.sqlite3`` file is the original trace generated by the C tracer and maintained in a SQLite database; it contains all the information about the experiment, in case the configuration file is insufficient in some aspect. This file is used, for instance, by the *graph* unpacker, so that it can recover the exact hierarchy of processes, together with the executable images they execute and the files they access (with the time and mode of these accesses).

.. seealso:: :ref:`Trace Database Schema <trace-schema>`

Structure
'''''''''

An unpacker is a Python module. It can be distributed separately or be part of a bigger distribution, given that it is declared in that distribution's ``setup.py`` as an `entry_point` to be registered with `pkg_resources` (see `setuptools' dynamic discovery of services and plugins <https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins>`__ section). You should declare a function as `entry_point` ``reprounzip.unpackers``. The name of the entry_point (the part before ``=``) will be the reprounzip subcommand, and the value is a callable that will get called with the :class:`argparse.ArgumentParser` object for that subcommand.
An unpacker is a Python module. It can be distributed separately or be part of a bigger distribution, given that it is declared in that distribution's ``setup.py`` as an `entry_point` to be registered with `pkg_resources` (see `setuptools' dynamic discovery of services and plugins <https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins>`__ section). You should declare a function as `entry_point` ``reprounzip.unpackers``. The name of the entry_point (before ``=``) will be the *reprounzip* subcommand, and the value is a callable that will get called with the :class:`argparse.ArgumentParser` object for that subcommand.

The package :mod:`reprounzip.unpackers` is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the ``__init__.py`` file (which you can copy from `here <https://github.com/ViDA-NYU/reprozip/blob/master/reprounzip/reprounzip/unpackers/__init__.py>`__) so namespace packages work correctly.
The package :mod:`reprounzip.unpackers` is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the ``__init__.py`` file (which you can copy from `here <https://github.com/ViDA-NYU/reprozip/blob/master/reprounzip/reprounzip/unpackers/__init__.py>`__) so that namespace packages work correctly.

The modules :mod:`reprounzip.common`, :mod:`reprounzip.utils`, and :mod:`reprounzip.unpackers.common` contain utilities that you might want to use (make sure to list reprounzip as a requirement in your ``setup.py``).
The modules :mod:`reprounzip.common`, :mod:`reprounzip.utils`, and :mod:`reprounzip.unpackers.common` contain utilities that you might want to use (make sure to list *reprounzip* as a requirement in your ``setup.py``).

Example of ``setup.py``::

Expand All @@ -54,14 +56,14 @@ Example of ``setup.py``::
Usual Commands
''''''''''''''

If possible, you should try to follow the same command names that the official unpackers use; these are:
If possible, you should try to follow the same command names that the official unpackers use, which are:

* ``setup``, to create the experiment directory and set everything for execution;
* ``run``, to reproduce the experiment;
* ``destroy``, to bring down all that setup and to prepare and delete the experiment directory safely;
* ``upload`` and ``download``, to either replace input files in the experiment with your own, or to get the output files for further examination.
* ``setup``: to create the experiment directory and set everything for execution;
* ``run``: to reproduce the experiment;
* ``destroy``: to bring down all that setup and to prepare and delete the experiment directory safely;
* ``upload`` and ``download``: to replace input files in the experiment, and to get the output files for further examination, respectively.

If these commands can be broken down into different steps that you want to expose to the user, or if you provide completely different actions from these defaults, you are free to add them to the parser as well. For instance, reprounzip-vagrant exposes ``setup/start``, which starts or resumes the virtual machine, and ``destroy/vm``, which stops and deallocates the virtual machine but leaves the template for possible reuse.
If these commands can be broken down into different steps that you want to expose to the user, or if you provide completely different actions from these defaults, you are free to add them to the parser as well. For instance, the *vagrant* unpacker exposes ``setup/start``, which starts or resumes the virtual machine, and ``destroy/vm``, which stops and deallocates the virtual machine but leaves the template for possible reuse.

A Note on File Paths
''''''''''''''''''''
Expand All @@ -77,7 +79,7 @@ Unpackers usually create a directory with everything necessary to later run the
$ reprounzip vagrant setup someexperiment.rpz mydirectory
$ reprounzip vagrant upload mydirectory /tmp/replace.txt:input_text

Unpackers unpack the config.yml file to the root of that directory, and keep status information in a ``.reprounzip`` file, which is a dict in :mod:`pickle` format. Following the same structure will allow the ``showfiles`` command, as well as :class:`~reprounzip.unpackers.common.FileUploader` and :class:`~reprounzip.unpackers.common.FileDownloader` classes, to work automatically. Please try to follow this structure.
Unpackers unpack the config.yml file to the root of that directory, and keep status information in a ``.reprounzip`` file, which is a dict in :mod:`pickle` format. Following the same structure will allow the ``showfiles`` command, as well as :class:`~reprounzip.unpackers.common.FileUploader` and :class:`~reprounzip.unpackers.common.FileDownloader` classes, to work correctly. Please try to follow this structure.

Signals
'''''''
Expand Down
43 changes: 21 additions & 22 deletions docs/source/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ For Linux distributions, both *reprozip* and *reprounzip* components are availab
Required Software Packages
--------------------------

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug]_. If you don't have it yet on your machine, you can get Python from `python.org <https://www.python.org/>`__ [#deb]_; you should prefer a 2.x releases to 3.x. You will also need the `pip <https://pip.pypa.io/en/latest/installing.html>`__ installer.
Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__ [#deb]_; you should prefer a 2.x release to a 3.x one. You will also need the `pip <https://pip.pypa.io/en/latest/installing.html>`__ installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

Expand All @@ -31,17 +31,16 @@ Besides Python and pip, each component or plugin to be used may have additional
+------------------------------+----------------------------------------------+
| *reprounzip-vagrant* | Python headers [#deb3]_ [#pycrypton]_, |
| | a working C compiler [#pycrypton]_, |
| | `Vagrant <https://www.vagrantup.com/>`__ at |
| | least version 1.1, |
| | `Vagrant 1.1+ <https://www.vagrantup.com/>`__|
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------------+----------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
+------------------------------+----------------------------------------------+

.. [#bug] ``reprounzip graph`` will not work due to `Python bug 13676 <http://bugs.python.org/issue13676>`__ related to sqlite3.
.. [#deb] On Debian and Debian-based, this is provided by *python*: ``sudo apt-get install python``.
.. [#deb2] On Debian and Debian-based, this is provided by *libsqlite3-dev*: ``sudo apt-get install libsqlite3-dev``.
.. [#deb3] On Debian and Debian-based, this is provided by *python-dev*: ``sudo apt-get install python-dev``.
.. [#deb] On Debian and Debian-based, you can use ``sudo apt-get install python``.
.. [#deb2] On Debian and Debian-based, you can use ``sudo apt-get install libsqlite3-dev``.
.. [#deb3] On Debian and Debian-based, you can use ``sudo apt-get install python-dev``.
.. [#pycrypton] Required to build `PyCrypto <https://www.dlitz.net/software/pycrypto/>`__.
Installing *reprozip*
Expand Down Expand Up @@ -84,27 +83,27 @@ For Mac OS X, only the *reprounzip* component is available.
Binaries
--------

An installer containing Python 2.7, *reprounzip* and all the plugins can be `downloaded from GitHub <https://github.com/ViDA-NYU/reprozip/releases/download/0.6/reprounzip-0.6.pkg>`__.
An installer containing Python 2.7, *reprounzip*, and all the plugins can be `downloaded from GitHub <https://github.com/ViDA-NYU/reprozip/releases/download/0.6/reprounzip-0.6.pkg>`__.

Required Software Packages
--------------------------

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug2]_. If you don't have it yet on your machine, you can get Python from `python.org <https://www.python.org/>`__; you should prefer a 2.x releases to 3.x. You will also need the pip installer (see `their installation instructions here <https://pip.pypa.io/en/latest/installing.html>`__).
Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug2]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__; you should prefer a 2.x release to a 3.x one. You will also need the `pip <https://pip.pypa.io/en/latest/installing.html>`__ installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

+------------------------------+----------------------------------------------+
| Component / Plugin | Required Software Packages |
+==============================+==============================================+
| *reprounzip* | None |
+------------------------------+----------------------------------------------+
| *reprounzip-vagrant* | Python headers [#macn]_ [#pycrypton2]_, |
| | a working C compiler [#macn]_ [#pycrypton2]_,|
| | `Vagrant <https://www.vagrantup.com/>`__, |
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------------+----------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
+------------------------------+----------------------------------------------+
+------------------------------+-----------------------------------------------+
| Component / Plugin | Required Software Packages |
+==============================+===============================================+
| *reprounzip* | None |
+------------------------------+-----------------------------------------------+
| *reprounzip-vagrant* | Python headers [#macn]_ [#pycrypton2]_, |
| | a working C compiler [#macn]_ [#pycrypton2]_, |
| | `Vagrant 1.1+ <https://www.vagrantup.com/>`__,|
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------------+-----------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
+------------------------------+-----------------------------------------------+

.. [#bug2] ``reprounzip graph`` will not work due to `Python bug 13676 <http://bugs.python.org/issue13676>`__ related to sqlite3.
.. [#macn] This is usually provided by installing Xcode (in the Mac App Store) and the Command Line Developer Tools; instructions on installing the latter may depend on your Mac OS X version (some information on StackOverflow `here <http://stackoverflow.com/questions/9329243/xcode-4-4-and-later-install-command-line-tools?answertab=active#tab-top>`__).
Expand Down Expand Up @@ -150,7 +149,7 @@ A 32-bit installer containing Python 2.7, *reprounzip*, and all the plugins can
Required Software Packages
--------------------------

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug3]_. If you don't have it yet on your machine, you can get Python from `python.org <https://www.python.org/>`__; you should prefer a 2.x releases to 3.x. You will also need the `pip <https://pip.pypa.io/en/latest/installing.html>`__ installer.
Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow *reprounzip* to work, but some features will not be available [#bug3]_. If you don't have Python on your machine, you can get it from `python.org <https://www.python.org/>`__; you should prefer a 2.x release to a 3.x one. You will also need the `pip <https://pip.pypa.io/en/latest/installing.html>`__ installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

Expand All @@ -160,7 +159,7 @@ Besides Python and pip, each component or plugin to be used may have additional
| *reprounzip* | None |
+------------------------------+------------------------------------------------------------------------+
| *reprounzip-vagrant* | `PyCrypto <https://www.dlitz.net/software/pycrypto/>`__ [#pycrypton3]_,|
| | `Vagrant <https://www.vagrantup.com/>`__, |
| | `Vagrant 1.1+ <https://www.vagrantup.com/>`__, |
| | `VirtualBox <https://www.virtualbox.org/>`__ |
+------------------------------+------------------------------------------------------------------------+
| *reprounzip-docker* | `Docker <https://www.docker.com/>`__ |
Expand Down

0 comments on commit 1bfa80a

Please sign in to comment.