Skip to content
Branch: master
Commits on Feb 21, 2020
  1. Merge branch 'fix-failing-master'

    kyleam committed Feb 21, 2020
Commits on Feb 20, 2020
  1. TST: travis: Install released version of DataLad

    kyleam committed Feb 20, 2020
    Our build that pulls in DataLad is failing because we use the
    deprecated `add` command, which has been removed entirely from
    DataLad's master [*].  Let's get back to a passing state by installing the
    latest release of DataLad.  We started installing DataLad's master in
    0a2c30f (TST: travis: Temporarily install developmental DataLad,
    2018-11-20) to get access to functionality that hadn't made it into a
    release yet.  That functionality is now in a release, so go back to
    installing the released version.
    [*] ReproMan needs an update for new DataLad functionality, which will
        include dropping our use of `add` and bumping our required DataLad
  2. MNT: Bump Python dependency to 3.5

    kyleam committed Feb 20, 2020
    The latest PyYAML release (5.3) has dropped support for Python 3.4,
    putting our 3.4 build on Travis into a failing state [0].  When we
    transitioned to Python 3 only, we didn't have a strong reason for
    choosing 3.4 as our floor (discussed in gh-369), so let's just bump it
    to 3.5.
  3. MNT: Set python_requires

    kyleam committed Feb 20, 2020
    This should have been done in gh-374 when we dropped Python 2 support.
Commits on Jan 3, 2020
  1. Merge pull request #497 from kyleam/jobs-no-local-dir

    kyleam committed Jan 3, 2020
  2. BF: jobs: Treat deleted local directory as orchestrator error

    kyleam committed Jan 2, 2020
    Trying to resurrect the orchestrator for a job where the local working
    directory has been deleted leads to a FileNotFoundError.  Recast this
    exception as an OrchestratorError so that (1) we can provide a more
    helpful message and (2) the jobs() call can continue with any other
    jobs specified rather than aborting.
Commits on Jan 2, 2020
  1. TST: run: Add local directory to context() fixture

    kyleam committed Jan 2, 2020
    The context() fixture calls run() from the top-level temporary
    directory that is set to $HOME.  This is a bit unusual because the
    local working directory for a run will almost certainly not be $HOME,
    but it's not currently problematic for any of the tests.
    The next commit, however, will delete the local working directory
    immediately following the run() call.  As is, that would delete
    $HOME/.reproman/run-root/ as part of the local working directory and
    leave behind the process for `/bin/sh [...]/runscript 0`.
    Move the local working directory to $HOME/local to avoid this issue.
  2. BF: jobs: Don't dump job spec when logging orchestrators errors

    kyleam committed Jan 2, 2020
    When we started logging orchestrator errors in ab1c577 (RF: jobs:
    Continue if action raises OrchestratorError, 2019-03-04), we put the
    entire job spec into the log message.  That provides a flood of
    information that is hard for the user to parse.  Replace the job spec
    with the job ID, which was presumably the intended value in the first
Commits on Dec 17, 2019
  1. Merge pull request #496 from chaselgrove/bf-remove-is-on-literal-comp…

    kyleam committed Dec 17, 2019
Commits on Dec 2, 2019
  1. DOC: execute: Clarify sentence about git-tracked inputs

    kyleam committed Dec 2, 2019
    f6d3647 (DOC: a few minor tweaks and fixes to execute.rst,
    2019-12-02) replaced "were" with "was", but that's incorrect if the
    sentence is describing the hypothetical case where the file it tracked
    by git-annex, not Git (which was the intention).  Rephrase the
    sentence to fix the grammar and avoid confusion.
    Re: #485 (review)
  2. DOC: execute: Consistently use code tags for file names

    kyleam committed Dec 2, 2019
    f6d3647 (DOC: a few minor tweaks and fixes to execute.rst,
    2019-12-02) put tags around one file name, so do the same for the
    other instance in execute.rst.
    Re: #485
Commits on Nov 19, 2019
  1. MNT: Update collections.Mapping for new location

    kyleam committed Nov 19, 2019
    As of Python 3.3, the ABCs in the collections module are available
    from  They remain exposed in `collections` for
    compatibility reasons but are slated for removal (originally in Python
    3.8, now in 3.9).
  2. ENH: docker: Pull :latest if no tag or digest is specified

    kyleam committed Nov 19, 2019
    If a tag or digest isn't specified when calling `reproman create`, we
    download all tagged images.  This behavior will likely be confusing to
    any callers that are familiar with the command-line interface to
    docker pull, which defaults to :latest if a tag or digest is omitted.
    And the behavior doesn't fit our goal: starting a new container from a
    single image.
    Teach the DockerContainer resource to append ":latest" when it
    receives a bare image name.
    Closes #483.
Commits on Nov 14, 2019
  1. DOC: Add section on command execution

    kyleam committed Nov 14, 2019
    The run functionality already contains a fair bit of documentation
    through `run --help, `run--list=`, and `jobs --help`.  Within
    execute.rst, provide a higher-level overview of the functionality
    while trying to avoid regurgitating much of the more focused
    information that is already provided by the above commands.
  2. ENH: setup: Add anchors to generated rst manpages

    kyleam committed Nov 14, 2019
    This allows us to refer to them with with ref:`reproman create
    As with most of ReproMan's setup, this came from DataLad, and this
    follows what DataLad currently does.  We should probably bring in its
    current setup wholesale via the 3rd branch.
  3. DOC: sphinx: Don't process sandbox/hierarchy.rst

    kyleam committed Nov 14, 2019
    This file isn't included in the html output but makes sphinx-build
    spew a lot of warnings.  Tell sphinx to ignore it.
  4. DOC: docs: Fix some typos

    kyleam committed Nov 14, 2019
Commits on Nov 8, 2019
  1. Merge pull request #481 from kyleam/docker-py-update

    kyleam committed Nov 8, 2019
Commits on Nov 5, 2019
  1. MNT: Unpin docker-py

    kyleam committed Nov 5, 2019
    The docker module hasn't been released under the name "docker-py"
    since 2016 [0] and has seen many releases under the name "docker" [1].
    So we've effectively been pinning the release to version 1.10.6 even
    though the current version is 4.1.0.
    To update to latest version, we need to substitute APIClient for
    Client (renamed in v2.3.0).  We also need to teach DockerSession.get()
    about the new return value of APIClient.get_archive(): as of v3.0.0,
    the content is returned as a generator of chunks rather than a byte
    Specify v3.0.0 as the new docker-py floor in
    Closes #387.
Commits on Oct 31, 2019
  1. NF: run: Extend --follow with optional stopping/deleting actions

    kyleam committed Oct 24, 2019
    Make it possible to stop (and optionally delete) a resource after
    within fetch() once we no longer need access to it.  This minimizes
    the amount of time that the resource if running after it's no longer
    This covers the proposed actions in gh-469 that are actually tied to
    --follow aside from emailing.
  2. TST: run: Convert resource manager from module to function scope

    kyleam committed Oct 31, 2019
    The next commit introduces fetch actions, one of which is deleting a
    resource.  To use this fixture there, we need to change the scope to
    "function" to that the deletion doesn't affect other runs.  If this
    ends up slowing down this module's tests too much, we can rework
    things so that there is both a module- and function-scoped fixture.
Commits on Oct 24, 2019
  1. RF: orchestrators: Add `failed` keyword argument to log_failed()

    kyleam committed Oct 24, 2019
    In an upcoming commit, fetch() will need to call get_failed_subjobs().
    This argument will allow it to feed log_failed() those results.
  2. RF: orchestrators: Convert failed_subjobs property to method

    kyleam committed Oct 24, 2019
    This property does a good amount of work underneath and depends on the
    resource.  We could put @cached_property on top, but a method probably
    makes the expense a little clearer.
  3. ENH: orchestrators: Do more fine-grained logging around fetch()

    kyleam committed Oct 24, 2019
    ... with the primary goal of making it clear when we are done working
    with the remote resource.
    Closes #470.
  4. RF: orchestrators: Move log_failed() calls earlier

    kyleam committed Oct 24, 2019
    These calls may require access to remote both to find out what jobs
    failed and to pull down relevant metadata files.  Call log_failed()
    before local processing that does not depend on the remote.  This will
    allow us to notify the caller mid-fetch that we're don't need access
    to the resource any longer.
Commits on Oct 10, 2019
  1. ENH: run: Version the dumped spec

    kyleam committed Oct 10, 2019
    As of 903b7aa (gh-464), we dump the spec into the metadata
    directory.  One of the reasons for doing this is to make it possible
    to use a dumped spec as the base for a spec fed back into the run().
    Version the spec so that if we introduce incompatible changes to the
    spec in the future, we can easily recognize old specs.
  2. ENH: run: Include reproman version in dumped spec

    kyleam committed Oct 10, 2019
    For debugging purposes, it can be useful to know which version of
    reproman (at least to the resolution of __version__) generated the
  3. Merge pull request #465 from kyleam/register-pytest-mark

    kyleam committed Oct 10, 2019
Commits on Oct 3, 2019
  1. TST: pytest: Register custom 'integration' mark

    kyleam committed Oct 3, 2019
    As of pytest 4.5.0, decorating tests with `pytest.mark.integration`
    results in this warning:
        PytestUnknownMarkWarning: Unknown pytest.mark.integration - is
        this a typo?  You can register custom marks to avoid this warning
        - for details, see
    Avoid it by registering 'integration' as a known marker.
Commits on Oct 2, 2019
  1. TST: orchestrators: Add rudimentary test for reusing dumped spec

    kyleam committed Oct 2, 2019
    The test is limited because our orchestrators currently don't deal
    well with existing files from the previous run in at least two ways:
    the session.copy() method will fail when trying to copy input files
    that already exist on a plain remote and, with a datalad remote,
    specified outputs aren't unlocked.  Until the second issue is
    resolved, we're only testing a case where we reset to the initial
    Also, note that this test is working at the orchestrator class level,
    so we don't yet have a check that interface/ can handle the
    dumped job spec.
  2. TST: orchestrators: Move check into a function

    kyleam committed Oct 2, 2019
    ... to prepare for testing a subsequent run using the dumped spec from
    the first.
  3. ENH: run: Use underscore for inaccessible job spec keys

    kyleam committed Sep 30, 2019
    The job spec contains keys that can be set by the user (through
    --job-parameter and --job-spec) as well as inaccessible keys that are
    set by us. The keys are now much more visible to the user now that we
    dump the job spec as a metadata file, so it's more important that we
    distinguish these two types of keys.  As a first step, let's prefix
    keys that are inaccessible with an underscore.
    We don't currently advertise that you can reuse the previous spec, but
    where ever we end up doing that we should mention that what the
    underscore prefix indicates.
    The inaccessible keys correspond to keys that we generate during the
    run, so we may want to just filter out keys that start with an
    underscore right when the job specs come into
You can’t perform that action at this time.