Merge branch 'master' into 'reprozip-web'

VIDA-NYU · Apr 27, 2023 · 8a830d7 · 8a830d7
2 parents 6bda4f6 + 04e2691
commit 8a830d7
Show file tree

Hide file tree

Showing 27 changed files with 352 additions and 109 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -67,13 +67,15 @@ jobs:
         os: [ubuntu-latest]
         mode: [tests]
         python:
-        - "3.6"
         - "3.7"
         - "3.8"
         include:
         - os: ubuntu-latest
           mode: coverage
           python: "3.9"
+        - os: ubuntu-20.04
+          mode: tests
+          python: "3.6"
     runs-on: ${{ matrix.os }}
     env:
       TEST_MODE: ${{ matrix.mode }}
@@ -82,10 +84,10 @@ jobs:
       LANG: C
       LC_ALL: C
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
       with:
         fetch-depth: 20
-    - uses: actions/setup-python@v1
+    - uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python }}
     - name: Install dependencies

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,6 +11,29 @@ Features:
 * Configuration file contains the walltime taken by each run
 * It is now possible to upload or download any file via its full path
 
+1.2.1 (2023-02-06)
+------------------
+
+(reprounzip and reprounzip-qt only)
+
+Bugfixes:
+* Fix typo in reprounzip, reprounzip-qt: import from reprounzip, not reprozip
+
+1.2 (2023-02-06)
+----------------
+
+Bugfixes:
+* Don't mark symlinks as input files
+* Fix reprounzip-vagrant not terminating after it says that it can't install packages
+* Add defense for CVE-2007-4559
+* Fix OrderedSet for Python 3.10+ compatibility
+
+Enhancements:
+* Recognize Ruby gems and apps and gather the whole environment
+* Don't mark Python .pth files as input files
+* Accept ZIP files in addition to TAR for RPZ files (reprozip doesn't currently create ZIP files)
+* Handle more Linux system calls: faccessat2, statx, execveat, clone3, openat2, fchownat, fchmodat, accept4, renameat2
+
 1.1 (2021-07-06)
 ----------------
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -30,7 +30,7 @@ Please follow the [GitHub Community Guidelines](https://docs.github.com/en/githu
 7. Commit your changes. From the command line:
  * `git add <FILE-NAMES>`
  * `git commit -m "A descriptive commit message"`
-8. While you were working some other changes might have gone in and break your stuff or vice versa. This can be a *merge conflict* but also conflicting behavior or code. Before you test, merge with master.
+8. While you were working some other changes might have gone in and break your stuff or vice versa. This can be a *merge conflict* but also conflicting behavior or code. Before you test, merge with upstream.
  * `git fetch upstream`
  * `git merge upstream/master`
 9. Test. Run the program and do something related to your feature/fix.

diff --git a/README.md b/README.md
@@ -2,8 +2,8 @@
 [![Coverage Status](https://codecov.io/github/VIDA-NYU/reprozip/coverage.svg?branch=master)](https://codecov.io/github/VIDA-NYU/reprozip?branch=master)
 [![Documentation Status](https://readthedocs.org/projects/reprozip/badge/?version=master)](https://docs.reprozip.org/en/master/)
 [![Matrix](https://img.shields.io/badge/chat-matrix.org-blue.svg)](https://riot.im/app/#/room/#reprozip:matrix.org)
-[![status](https://img.shields.io/badge/JOSS-10.21105%2Fjoss.00107-green.svg)](http://joss.theoj.org/papers/b578b171263c73f64dfb9d040ca80fe0)
-[![DOI](https://img.shields.io/badge/DOI/10.5281%2Fzenodo.5081097-green.svg)](https://doi.org/10.5281/zenodo.5081097)
+[![paper](https://img.shields.io/badge/JOSS-10.21105%2Fjoss.00107-green.svg)](http://joss.theoj.org/papers/b578b171263c73f64dfb9d040ca80fe0)
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.7696658-green.svg)](https://doi.org/10.5281/zenodo.7696658)
 
 ReproZip
 ========
@@ -104,15 +104,6 @@ Bugs and feature plannings are tracked in the [GitHub issues](https://github.com
 
 To suggest changes to this source code, feel free to raise a [GitHub pull request](https://github.com/VIDA-NYU/reprozip/pulls).  Any contributions received are assumed to be covered by the [BSD 3-Clause license](LICENSE.txt). We might ask you to sign a _Contributor License Agreement_ before accepting a larger contribution.
 
-Research notice
----------------
-
-Please note that this repository is participating in a study into sustainability of open source projects. Data will be gathered about this repository for approximately the next 12 months, starting from June 2021.
-
-Data collected will include number of contributors, number of PRs, time taken to close/merge these PRs, and issues closed.
-
-For more information, please visit [the informational page](https://sustainable-open-science-and-software.github.io/) or download the [participant information sheet](https://sustainable-open-science-and-software.github.io/assets/PIS_sustainable_software.pdf).
-
 License
 -------
 

diff --git a/docs/developerguide.rst b/docs/developerguide.rst
@@ -6,7 +6,7 @@ Developer's Guide
 General Development Information
 -------------------------------
 
-Development happens on `GitHub <https://github.com/ViDA-NYU/reprozip>`__; bug reports and feature requests are welcome. If you are interested in giving us a hand, please do not hesitate to submit a pull request there.
+Development happens on `GitHub <https://github.com/VIDA-NYU/reprozip>`__; bug reports and feature requests are welcome. If you are interested in giving us a hand, please do not hesitate to submit a pull request there.
 
 Continuous testing is provided by `GitHub Actions <https://github.com/VIDA-NYU/reprozip/actions>`__. Note that ReproZip still tries to support Python 2 as well as Python 3. Test coverage is not very high because there are a lot of operations that are difficult to cover on CI (for instance, Vagrant VMs cannot be used over there).
 
@@ -48,9 +48,9 @@ The ``METADATA/trace.sqlite3`` file is the original trace generated by the C tra
 Structure
 '''''''''
 
-An unpacker is a Python module. It can be distributed separately or be a part of a bigger distribution, given that it is declared in that distribution's ``setup.py`` as an `entry_point` to be registered with `pkg_resources` (see `setuptools' advertising behavior section <https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#advertising-behavior>`__). You should declare a function as `entry_point` ``reprounzip.unpackers``. The name of the entry_point (before ``=``) will be the *reprounzip* subcommand, and the value is a callable that will get called with the :class:`argparse.ArgumentParser` object for that subcommand.
+An unpacker is a Python module. It can be distributed separately or be a part of a bigger distribution, given that it is declared in that distribution's ``setup.py`` as an `entry_point` to be registered with `pkg_resources` (see `setuptools' advertising behavior section <https://setuptools.pypa.io/en/latest/userguide/entry_point.html#advertising-behavior>`__). You should declare a function as `entry_point` ``reprounzip.unpackers``. The name of the entry_point (before ``=``) will be the *reprounzip* subcommand, and the value is a callable that will get called with the :class:`argparse.ArgumentParser` object for that subcommand.
 
-The package :mod:`reprounzip.unpackers` is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the ``__init__.py`` file (which you can copy from `here <https://github.com/ViDA-NYU/reprozip/blob/master/reprounzip/reprounzip/unpackers/__init__.py>`__) so that namespace packages work correctly.
+The package :mod:`reprounzip.unpackers` is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the ``__init__.py`` file (which you can copy from `here <https://github.com/VIDA-NYU/reprozip/blob/master/reprounzip/reprounzip/unpackers/__init__.py>`__) so that namespace packages work correctly.
 
 The modules :mod:`reprounzip.common`, :mod:`reprounzip.utils`, and :mod:`reprounzip.unpackers.common` contain utilities that you might want to use (make sure to list *reprounzip* as a requirement in your ``setup.py``).
 
@@ -99,7 +99,7 @@ Unpackers unpack the config.yml file to the root of that directory, and keep sta
 Signals
 '''''''
 
-Since version 0.4.1, `reprounzip` has signals that can be used to hook in plugins, although no such plugin has been released at this time. To ensure that these work correctly when using your unpacker, you should emit them when appropriate. The complete list of signals is available in `signal.py <https://github.com/ViDA-NYU/reprozip/blob/master/reprounzip/reprounzip/signals.py>`__.
+Since version 0.4.1, `reprounzip` has signals that can be used to hook in plugins, although no such plugin has been released at this time. To ensure that these work correctly when using your unpacker, you should emit them when appropriate. The complete list of signals is available in `signal.py <https://github.com/VIDA-NYU/reprozip/blob/master/reprounzip/reprounzip/signals.py>`__.
 
 Final Observations
 ------------------

diff --git a/docs/index.rst b/docs/index.rst
@@ -36,5 +36,5 @@ Links
 -----
 
 * `Project website <https://www.reprozip.org/>`__
-* `GitHub repository <https://github.com/ViDA-NYU/reprozip>`__
+* `GitHub repository <https://github.com/VIDA-NYU/reprozip>`__
 * Mailing list: `reprozip@nyu.edu <https://groups.google.com/a/nyu.edu/g/reprozip>`__
diff --git a/docs/install.rst b/docs/install.rst
@@ -4,7 +4,7 @@ Installation
 ************
 
 ReproZip is available as open source, released under the Revised BSD License. The tool is comprised of two components: **reprozip** (for the packing step) and **reprounzip** (for the unpack step). Additional components and plugins are also provided for *reprounzip*: **reprounzip-vagrant**, which unpacks the experiment in a Vagrant virtual machine; **reprounzip-docker**, which unpacks the experiment in a Docker container; and **reprounzip-vistrails**, which creates a VisTrails workflow to reproduce the experiment. More plugins may be developed in the future (and, of course, you are free to :ref:`roll your own <develop-plugins>`).
-In our `website <https://www.reprozip.org/>`__, you can find links to our PyPI packages and our `GitHub repository <https://github.com/ViDA-NYU/reprozip>`__.
+In our `website <https://www.reprozip.org/>`__, you can find links to our PyPI packages and our `GitHub repository <https://github.com/VIDA-NYU/reprozip>`__.
 
 In the following, you will find installation instructions for :ref:`linux`, :ref:`mac`, and :ref:`windows`. ReproZip is also available for the :ref:`conda` Python distribution.
 
@@ -46,14 +46,14 @@ Debian and Ubuntu
 
 You can get all the required dependencies using APT::
 
-    apt-get install python python-dev python-pip gcc libsqlite3-dev libssl-dev libffi-dev
+    apt-get install python3 python3-dev python3-pip gcc libsqlite3-dev libssl-dev libffi-dev
 
 Fedora & CentOS
 ```````````````
 
 You can get the dependencies using the Yum packaging manager::
 
-    yum install python python-devel gcc sqlite-devel openssl-devel libffi-devel
+    yum install python3 python3-devel gcc sqlite-devel openssl-devel libffi-devel
 
 ..  [#bug] ``reprozip`` and ``reprounzip graph`` will not work before 2.7.3 due to `Python bug 13676 <https://bugs.python.org/issue13676>`__ related to sqlite3. Python 2.6 is ancient and unsupported.
 ..  [#vis1] `VisTrails v2.2.3+ <https://www.vistrails.org/>`__ is required to run the workflow generated by the plugin.
@@ -167,7 +167,7 @@ Besides Python and pip, each component or plugin to be used may have additional
 
 ..  [#bug3] ``reprozip`` and ``reprounzip graph`` will not work before 2.7.3 due to `Python bug 13676 <https://bugs.python.org/issue13676>`__ related to sqlite3. Python 2.6 is ancient and unsupported.
 ..  [#vis3] `VisTrails v2.2.3+ <https://www.vistrails.org/>`__ is required to run the workflow generated by the plugin.
-..  [#windowshome] Windows Professional Edition is required for Docker, it will not work on Windows Home Edition; `see FAQ <https://docs.docker.com/desktop/faqs/#can-i-install-docker-desktop-on-windows-10-home>`__.
+..  [#windowshome] Windows Professional Edition is required for Docker, it will not work on Windows Home Edition; `see FAQ <https://docs.docker.com/desktop/faqs/windowsfaqs/#can-i-install-docker-desktop-on-windows-10-home>`__.
 
 Installing *reprounzip*
 -----------------------
@@ -186,7 +186,7 @@ Or you can install *reprounzip* and choose components manually::
 Anaconda
 ========
 
-*reprozip* and *reprounzip* can also be installed on the `Anaconda <https://www.anaconda.com/products/individual#Downloads>`__ Python distribution, from anaconda.org::
+*reprozip* and *reprounzip* can also be installed on the `Anaconda <https://www.anaconda.com/products/distribution#Downloads>`__ Python distribution, from anaconda.org::
 
     $ conda install --channel conda-forge reprozip reprounzip reprounzip-docker reprounzip-vagrant reprounzip-vistrails
 

diff --git a/reprounzip-docker/README.rst b/reprounzip-docker/README.rst
@@ -1,5 +1,5 @@
-ReproZip
-========
+ReproZip project
+================
 
 `ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a bundle that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).  A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
 

diff --git a/reprounzip-qt/README.rst b/reprounzip-qt/README.rst
@@ -1,5 +1,5 @@
-ReproZip
-========
+ReproZip project
+================
 
 `ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a bundle that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).  A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
 

diff --git a/reprounzip-vagrant/README.rst b/reprounzip-vagrant/README.rst
@@ -1,5 +1,5 @@
-ReproZip
-========
+ReproZip project
+================
 
 `ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a bundle that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).  A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
 

diff --git a/reprounzip-vagrant/reprounzip_vagrant/__init__.py b/reprounzip-vagrant/reprounzip_vagrant/__init__.py
@@ -280,6 +280,7 @@ def vagrant_setup_create(args):
             logger.error("Need to install %d packages but couldn't select a "
                          "package installer: %s",
                          len(packages), e)
+            sys.exit(1)
 
     try:
         # Writes setup script

diff --git a/reprounzip-vistrails/README.rst b/reprounzip-vistrails/README.rst
@@ -1,5 +1,5 @@
-ReproZip
-========
+ReproZip project
+================
 
 `ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a bundle that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).  A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
 

diff --git a/reprounzip/README.rst b/reprounzip/README.rst
@@ -1,5 +1,5 @@
-ReproZip
-========
+ReproZip project
+================
 
 `ReproZip <https://www.reprozip.org/>`__ is a tool aimed at simplifying the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science. It tracks operating system calls and creates a bundle that contains all the binaries, files and dependencies required to run a given command on the author's computational environment (packing step).  A reviewer can then extract the experiment in his environment to reproduce the results (unpacking step).
 

diff --git a/reprounzip/reprounzip/orderedset.py b/reprounzip/reprounzip/orderedset.py
@@ -21,10 +21,13 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 
-import collections
+try:
+    from collections import MutableSet
+except ImportError:
+    from collections.abc import MutableSet
 
 
-class OrderedSet(collections.MutableSet):
+class OrderedSet(MutableSet):
     def __init__(self, iterable=None):
         self.end = end = []
         end += [None, end, end]         # sentinel node for doubly linked list

diff --git a/reprounzip/reprounzip/unpackers/common/misc.py b/reprounzip/reprounzip/unpackers/common/misc.py
@@ -468,7 +468,7 @@ def pty_spawn(*args, **kwargs):
     return pty.spawn(*args, **kwargs)
 
 
-def interruptible_call(cmd, **kwargs):
+def interruptible_call(cmd, request_tty=False, shell=False):
     assert signal.getsignal(signal.SIGINT) == signal.default_int_handler
     proc = [None]
 
@@ -482,7 +482,7 @@ def _sigint_handler(signum, frame):
     signal.signal(signal.SIGINT, _sigint_handler)
 
     try:
-        if kwargs.pop('request_tty', False):
+        if request_tty:
             try:
                 import pty  # noqa: F401
             except ImportError:
@@ -491,14 +491,14 @@ def _sigint_handler(signum, frame):
                 if hasattr(sys.stdin, 'isatty') and not sys.stdin.isatty():
                     logger.info("We need a tty and we are not attached to "
                                 "one. Opening pty...")
-                    if kwargs.pop('shell', False):
+                    if shell:
                         if not isinstance(cmd, (str, str)):
                             raise TypeError("shell=True but cmd is not a "
                                             "string")
                         cmd = ['/bin/sh', '-c', cmd]
                     res = pty_spawn(cmd)
                     return res >> 8 - (res & 0xFF)
-        proc[0] = subprocess.Popen(cmd, **kwargs)
+        proc[0] = subprocess.Popen(cmd, shell=shell)
         return proc[0].wait()
     finally:
         signal.signal(signal.SIGINT, signal.default_int_handler)

diff --git a/reprounzip/reprounzip/unpackers/default.py b/reprounzip/reprounzip/unpackers/default.py
@@ -74,6 +74,7 @@ def installpkgs(args):
         installer = select_installer(pack, runs)
     except CantFindInstaller as e:
         logger.error("Couldn't select a package installer: %s", e)
+        sys.exit(1)
 
     if args.summary:
         # Print out a list of packages with their status
@@ -244,7 +245,7 @@ def directory_run(args):
         cmd = 'cd %s && ' % shell_escape(
             str(join_root(root, Path(run['workingdir']))))
         cmd += '/usr/bin/env -i '
-        cmd += 'LD_LIBRARY_PATH=%s' % ':'.join(
+        cmd += 'LD_LIBRARY_PATH=%s ' % ':'.join(
             shell_escape(str(join_root(root, d)))
             for d in lib_dirs
         )
@@ -285,6 +286,10 @@ def directory_run(args):
         if cmdline is None:
             argv = run['argv']
 
+            # If the command is not a path, use the path instead
+            if '/' not in argv[0]:
+                argv = [run['binary']] + argv[1:]
+
             # Rewrites command-line arguments that are absolute filenames
             rewritten = False
             for i in range(len(argv)):