Skip to content

Commit

Permalink
Merge branch '1.0.x' into 'master'
Browse files Browse the repository at this point in the history
  • Loading branch information
@remram44
remram44 committed Jul 18, 2017
2 parents d665c49 + a5780c1 commit e113489
Show file tree
Hide file tree
Showing 20 changed files with 850 additions and 164 deletions.
7 changes: 5 additions & 2 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ Features:
* Configuration file contains the walltime taken by each run
* It is now possible to upload or download any file via its full path

1.0.10 (???)
------------
1.0.10 (2017-07-10)
-------------------

Bugfixes:
* Correctly escape shell commands containing backticks
Expand All @@ -23,6 +23,9 @@ Bugfixes:
* Using `graph --packages drop` together with `--json` no longer crashes

Enhancements:
* New reprozip-jupyter tool to trace and reproduce Jupyter notebooks
* reprozip_jupyter can be registered as a Jupyter extension to trace notebooks from the Jupyter web interface
* The Qt GUI knows to run packages with reprozip-jupyter if they are notebook environments (kernels) unpacked with Docker (and reprozip-jupyter is installed)
* Add `--docker-cmd` to reprounzip-docker to select the Docker command (for example `--docker-cmd="sudo docker"`)
* Implement `--expose-port` option for Vagrant and Docker (no need for `--docker-option=-p...`)
* Add docker-machine support to GUI (select which machine to use)
Expand Down
101 changes: 24 additions & 77 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,9 @@
ReproZip
========

[ReproZip][web] is a tool aimed at simplifying the process of creating
reproducible experiments from command-line executions, a frequently-used common
denominator in computational science.
[ReproZip][web] is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.

It tracks operating system calls and creates a package that contains all the
binaries, files and dependencies required to run a given command on the
author's computational environment (packing step). A reviewer can then extract
the experiment in his environment to reproduce the results (unpacking step).
It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step). A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).

Quickstart
----------
Expand All @@ -25,107 +20,72 @@ We have an [example repository](https://github.com/ViDA-NYU/reprozip-examples) w

### Packing

Packing experiments is only available for Linux distributions. In the
environment where the experiment is originally executed, first install
reprozip:
Packing experiments is only available for Linux distributions. In the environment where the experiment is originally executed, first install reprozip:

$ pip install reprozip

Then, run your experiment with reprozip. Suppose you execute your experiment by
originally running the following command:
Then, run your experiment with reprozip. Suppose you execute your experiment by originally running the following command:

$ ./myexperiment -my --options inputs/somefile.csv other_file_here.bin

To run it with reprozip, you just need to use the prefix *reprozip trace*:

$ reprozip trace ./myexperiment -my --options inputs/somefile.csv other_file_here.bin

This command creates a *.reprozip* directory, in which you'll find the
configuration file, named *config.yml*. You can edit the command line and
environment variables, and choose which files to pack.
This command creates a *.reprozip* directory, in which you'll find the configuration file, named *config.yml*. You can edit the command line and environment variables, and choose which files to pack.

If you are using Debian or Ubuntu, most of these files (library dependencies)
are organized by package. You can add or remove files, or choose not to include
a package by changing option *packfiles* from true to false. In this way,
smaller packs can be created with reprozip (if space is an issue), and
reprounzip can download these files from the package manager; however, note
this is only available for Debian and Ubuntu for now, and also be aware that
package versions might differ. Choosing which files to pack is also important
to remove sensitive information and third-party software that is not open
source and should not be distributed.
If you are using Debian or Ubuntu, most of these files (library dependencies) are organized by package. You can add or remove files, or choose not to include a package by changing option *packfiles* from true to false. In this way, smaller packs can be created with reprozip (if space is an issue), and reprounzip can download these files from the package manager; however, note this is only available for Debian and Ubuntu for now, and also be aware that package versions might differ. Choosing which files to pack is also important to remove sensitive information and third-party software that is not open source and should not be distributed.

Once done editing the configuration file (or even if you did not change
anything), run the following command to create a ReproZip package named
*my_experiment*:
Once done editing the configuration file (or even if you did not change anything), run the following command to create a ReproZip package named *my_experiment*:

$ reprozip pack my_experiment.rpz

Voilà! Now your experiment has been packed, and you can send it to your
collaborators, reviewers, and researchers around the world!
Voilà! Now your experiment has been packed, and you can send it to your collaborators, reviewers, and researchers around the world!

Note that you can open the help message for any reprozip command by using the
flag *-h*.
Note that you can open the help message for any reprozip command by using the flag *-h*.

### Unpacking

Do you need to unpack an experiment in a Linux machine? Easy! First, install
reprounzip:
Do you need to unpack an experiment in a Linux machine? Easy! First, install reprounzip:

$ pip install reprounzip

Then, if you want to unpack everything in a single directory named
*mydirectory* and execute the experiment from there, use the prefix *reprounzip
directory*:
Then, if you want to unpack everything in a single directory named *mydirectory* and execute the experiment from there, use the prefix *reprounzip directory*:

$ reprounzip directory setup my_experiment.rpz mydirectory
$ reprounzip directory run mydirectory

In case you prefer to build a chroot environment under *mychroot*, use the
prefix *reprounzip chroot*:
In case you prefer to build a chroot environment under *mychroot*, use the prefix *reprounzip chroot*:

$ reprounzip chroot setup my_experiment.rpz mychroot
$ reprounzip chroot run mychroot

Note that the previous options do not interfere with the original configuration
of the environment, so don't worry! If you are using Debian or Ubuntu,
reprounzip also has an option to install all the library dependencies directly
on the machine using package managers (rather than just copying the files from
the .rpz package). Be aware that this will interfere in your environment and it
may update your library packages, so use it at your own risk! For this option,
just use the prefix *reprounzip installpkgs*:
Note that the previous options do not interfere with the original configuration of the environment, so don't worry! If you are using Debian or Ubuntu, reprounzip also has an option to install all the library dependencies directly on the machine using package managers (rather than just copying the files from the .rpz package). Be aware that this will interfere in your environment and it may update your library packages, so use it at your own risk! For this option, just use the prefix *reprounzip installpkgs*:

$ reprounzip installpkgs my_experiment.rpz

What if you want to reproduce the experiment in Windows or Mac OS X? You can
build a virtual machine with the experiment! Easy as well! First, install the
plugin reprounzip-vagrant:
What if you want to reproduce the experiment in Windows or Mac OS X? You can build a virtual machine with the experiment! Easy as well! First, install the plugin reprounzip-vagrant:

$ pip install reprounzip-vagrant

Note that (i) you must install reprounzip first, and (ii) the plugin requires
having [Vagrant][vagrant] installed. Then, use the prefix *reprounzip vagrant*
to create and start a virtual machine under directory *mytemplate*:
Note that (i) you must install reprounzip first, and (ii) the plugin requires having [Vagrant][vagrant] installed. Then, use the prefix *reprounzip vagrant* to create and start a virtual machine under directory *mytemplate*:

$ reprounzip vagrant setup my_experiment.rpz mytemplate

To execute the experiment, simply run:

$ reprounzip vagrant run mytemplate

Alternatively, you may use [Docker][docker] containers to reproduce the
experiment, which also works under Linux, Mac OS X, and Windows! First, install
the plugin reprounzip-docker:
Alternatively, you may use [Docker][docker] containers to reproduce the experiment, which also works under Linux, Mac OS X, and Windows! First, install the plugin reprounzip-docker:

$ pip install reprounzip-docker

Then, assuming that you want to create the container under directory
*mytemplate*, simply use the prefix *reprounzip docker*:
Then, assuming that you want to create the container under directory *mytemplate*, simply use the prefix *reprounzip docker*:

$ reprounzip docker setup my_experiment.rpz mytemplate
$ reprounzip docker run mytemplate

Remember that you can open the help message and learn more about other
available flags and options by using the flag *-h* for any reprounzip command.
Remember that you can open the help message and learn more about other available flags and options by using the flag *-h* for any reprounzip command.

Citing ReproZip
---------------
Expand All @@ -139,38 +99,25 @@ Please use the following when citing ReproZip:
Contribute
----------

Please subscribe to and contact the
[reprozip-users](https://vgc.poly.edu/mailman/listinfo/reprozip-users) mailing
list for questions, suggestions and discussions about using reprozip.
Please subscribe to and contact the [reprozip-users](https://vgc.poly.edu/mailman/listinfo/reprozip-users) mailing list for questions, suggestions and discussions about using reprozip.

Use the [reprozip-dev](https://vgc.poly.edu/mailman/listinfo/reprozip-dev)
mailing list for any questions about reprozip source code.
Use the [reprozip-dev](https://vgc.poly.edu/mailman/listinfo/reprozip-dev) mailing list for any questions about reprozip source code.

Bugs and feature plannings are tracked in the
[GitHub issues](https://github.com/ViDA-NYU/reprozip/issues). Feel free to add
an issue!

To suggest changes to this source code, feel free to raise a
[GitHub pull request](https://github.com/ViDA-NYU/reprozip/pulls).
Any contributions received are assumed to be covered by the
[BSD 3-Clause license](LICENSE.txt). We might ask you to sign a
_Contributor License Agreement_ before accepting a larger contribution.
Bugs and feature plannings are tracked in the [GitHub issues](https://github.com/ViDA-NYU/reprozip/issues). Feel free to add an issue!

To suggest changes to this source code, feel free to raise a [GitHub pull request](https://github.com/ViDA-NYU/reprozip/pulls). Any contributions received are assumed to be covered by the [BSD 3-Clause license](LICENSE.txt). We might ask you to sign a _Contributor License Agreement_ before accepting a larger contribution.

License
-------

* Copyright (C) 2014-2017, New York University

Licensed under a **BSD 3-Clause license**. See the file
[LICENSE.txt](LICENSE.txt) for details.

Licensed under a **BSD 3-Clause license**. See the file [LICENSE.txt](LICENSE.txt) for details.

Links and References
--------------------

For more detailed information, please refer to our [website][web], as well as
to our [documentation][docs].
For more detailed information, please refer to our [website][web], as well as to our [documentation][docs].

ReproZip is currently being developed at [NYU][nyu]. The team includes:

Expand Down
20 changes: 3 additions & 17 deletions reprounzip-docker/README.rst
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,17 @@
ReproZip
========

`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible
experiments from command-line executions, a frequently-used common denominator
in computational science. It tracks operating system calls and creates a package
that contains all the binaries, files and dependencies required to run a given
command on the author's computational environment (packing step).
A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step). A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).

reprounzip-docker
-----------------

This is the component responsible for the unpacking step
on different environments (Linux, Windows, and Mac OS X)
by using a `Docker <https://www.docker.com/>`_ container.

Please refer to `reprozip <https://pypi.python.org/pypi/reprozip>`__,
`reprounzip <https://pypi.python.org/pypi/reprounzip>`_,
and `reprounzip-vagrant <https://pypi.python.org/pypi/reprounzip-vagrant>`_
for other components and plugins.

This is the component responsible for the unpacking step on different environments (Linux, Windows, and Mac OS X) by using a `Docker <https://www.docker.com/>`_ container. Please refer to `reprozip <https://pypi.python.org/pypi/reprozip>`__, `reprounzip <https://pypi.python.org/pypi/reprounzip>`_, and `reprounzip-vagrant <https://pypi.python.org/pypi/reprounzip-vagrant>`_ for other components and plugins.

Additional Information
----------------------

For more detailed information, please refer to our `website <https://www.reprozip.org/>`_, as well as to
our `documentation <https://reprozip.readthedocs.io/>`_.
For more detailed information, please refer to our `website <https://www.reprozip.org/>`_, as well as to our `documentation <https://reprozip.readthedocs.io/>`_.

ReproZip is currently being developed at `NYU <http://engineering.nyu.edu/>`_. The team includes:

Expand Down
13 changes: 3 additions & 10 deletions reprounzip-qt/README.rst
View file
Original file line number Diff line number Diff line change
@@ -1,26 +1,19 @@
ReproZip
========

`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible
experiments from command-line executions, a frequently-used common denominator
in computational science. It tracks operating system calls and creates a package
that contains all the binaries, files and dependencies required to run a given
command on the author's computational environment (packing step).
A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
`ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a package that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step). A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).

reprounzip-qt
-------------

This is a graphical user interface for the reprounzip package, allowing to unpack and run packed applications without having to use the command-line interface.

Please refer to `reprozip <https://pypi.python.org/pypi/reprozip>`__ and `reprounzip <https://pypi.python.org/pypi/reprounzip-vagrant>`_ for more information.

Please refer to `reprozip <https://pypi.python.org/pypi/reprozip>`__ and `reprounzip <https://pypi.python.org/pypi/reprounzip>`_ for more information.

Additional Information
----------------------

For more detailed information, please refer to our `website <https://www.reprozip.org/>`_, as well as to
our `documentation <https://reprozip.readthedocs.io/>`_.
For more detailed information, please refer to our `website <https://www.reprozip.org/>`_, as well as to our `documentation <https://reprozip.readthedocs.io/>`_.

ReproZip is currently being developed at `NYU <http://engineering.nyu.edu/>`_. The team includes:

Expand Down
23 changes: 20 additions & 3 deletions reprounzip-qt/reprounzip_qt/gui/run.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -291,15 +291,22 @@ def __init__(self, unpacked_directory='', **kwargs):
self.root.addItems(ROOT.TEXT)
layout.addWidget(self.root, 6, 1, 1, 2)

layout.addWidget(QtGui.QLabel("Jupyter integration:"),
7, 0)
self.run_jupyter_notebook = QtGui.QCheckBox("Run notebook server",
checked=False,
enabled=False)
layout.addWidget(self.run_jupyter_notebook, 7, 1, 1, 2)

group = QtGui.QGroupBox(title="Unpacker options")
group_layout = QtGui.QVBoxLayout()
self.unpacker_options = ResizableStack()
scroll = QtGui.QScrollArea(widgetResizable=True)
scroll.setWidget(self.unpacker_options)
group_layout.addWidget(scroll)
group.setLayout(group_layout)
layout.addWidget(group, 7, 0, 1, 3)
layout.setRowStretch(7, 1)
layout.addWidget(group, 8, 0, 1, 3)
layout.setRowStretch(8, 1)

for i, (name, WidgetClass) in enumerate(self.UNPACKERS):
widget = WidgetClass()
Expand All @@ -317,7 +324,7 @@ def __init__(self, unpacked_directory='', **kwargs):
self.destroy_widget = QtGui.QPushButton("Destroy unpacked experiment")
self.destroy_widget.clicked.connect(self._destroy)
buttons.addWidget(self.destroy_widget)
layout.addLayout(buttons, 8, 0, 1, 3)
layout.addLayout(buttons, 9, 0, 1, 3)

self.setLayout(layout)

Expand All @@ -338,6 +345,9 @@ def _directory_changed(self, new_dir=None, force=False):

unpacker = reprounzip.check_directory(self.directory)

self.run_jupyter_notebook.setChecked(False)
self.run_jupyter_notebook.setEnabled(False)

self.runs_widget.clear()
if unpacker is not None:
with open(self.directory + '/config.yml') as fp:
Expand All @@ -354,6 +364,12 @@ def _directory_changed(self, new_dir=None, force=False):
self.unpacker_options.setCurrentIndex(
dict((n, i) for i, (n, w) in enumerate(self.UNPACKERS))
.get(unpacker, 4))

if (unpacker == 'docker' and
reprounzip.find_command('reprozip-jupyter') is not None and
reprounzip.is_jupyter(self.directory)):
self.run_jupyter_notebook.setEnabled(True)
self.run_jupyter_notebook.setChecked(True)
else:
self.run_widget.setEnabled(False)
self.destroy_widget.setEnabled(False)
Expand All @@ -374,6 +390,7 @@ def _run(self):
self.directory, runs=runs,
unpacker=self.unpacker,
root=ROOT.INDEX_TO_OPTION[self.root.currentIndex()],
jupyter=self.run_jupyter_notebook.isChecked(),
**options))

def _destroy(self):
Expand Down

0 comments on commit e113489

Please sign in to comment.