Skip to content
Branch: master
Commits on Mar 22, 2019
  1. Merge pull request #398 from yarikoptic/opt-imports-lazy

    kyleam committed Mar 22, 2019
  2. TST: verify no imports of heavy modules upon import of .api

    yarikoptic committed Mar 22, 2019
Commits on Mar 21, 2019
  1. OPT: delay imports of requests, jinja2, paramiko (et al) until used

    yarikoptic committed Mar 21, 2019
    Otherwise it affects our startup time due to import of all the
    See #397
    for a more generic description of the issue and possible remedies.
  2. Merge pull request #395 from kyleam/silence-paramiko

    yarikoptic committed Mar 21, 2019
    MNT: ssh: Silence paramiko warnings
  3. MNT: ssh: Silence paramiko warnings

    kyleam committed Mar 21, 2019
    Using paramiko currently floods our output with warnings like
      CryptographyDeprecationWarning: encode_point has been deprecated on
      EllipticCurvePublicNumbers and will be removed in a future
      version. [...]
    Silence these because there's nothing for us to do here besides wait
    for a paramiko release that includes a fix.  Leave a comment pointing
    to the currently open paramiko PR that deals with these warnings.
    Closes #390.
Commits on Mar 18, 2019
  1. Merge pull request #331 from kyleam/nf-run

    yarikoptic committed Mar 18, 2019
    NF: run command
Commits on Mar 14, 2019
  1. Merge remote-tracking branch 'origin/master' into nf-run

    kyleam committed Mar 14, 2019
    Resolve conflicts for and by taking changes
    from both sides.
Commits on Mar 7, 2019
  1. Merge pull request #378 from kyleam/venv-pyc

    yarikoptic committed Mar 7, 2019
    retrace: Map pyc files back to their source
  2. Merge pull request #389 from kyleam/doc-ssh-pastos

    mjtravers committed Mar 7, 2019
    DOC: ssh: Clean up misplaced AWS-related bits
  3. DOC: ssh: Clean up misplaced AWS-related bits

    kyleam committed Mar 7, 2019
    These are presumably pastos from when the ssh resource was created.
  4. DOC: run: Point users to 'jobs' subcommand

    kyleam committed Mar 7, 2019
  5. DOC: orchestrators: Describe orchestrators in more detail

    kyleam committed Mar 7, 2019
    This will be available in the `run --list` output.  This might be a
    bit too verbose and some of it may belong elsewhere, but it's a start.
  6. ENH: run --list: Don't collapse multiple paragraphs

    kyleam committed Mar 7, 2019
    We should expand the description of the orchestrators and submitters,
    but to do that we'll need multiple paragraphs, and collapsing the
    paragraphs when rendering them would hurt readability.
  7. DOC: run: Add to-do comment

    kyleam committed Mar 7, 2019
  8. RF: run: Change short key for --job-parameter

    kyleam committed Mar 7, 2019
    "b" has no clear link to "--job-parameter". (I'm not even sure why it
    was used in the first place; I'm guessing it was something to do with
    create's "-b -> --backend-parameters".)
  9. BF: orchestrators: Push unique ref to local "remote" dataset

    kyleam committed Mar 6, 2019
    This code is just for local testing and isn't very clever (e.g., it
    doesn't handle non-fast-forward cases, which are easy to get into).
    Make it work a bit more reliably by pushing to a unique ref for each
Commits on Mar 6, 2019
  1. BF: orchestrators: Push ref when creating sibling on local shell

    kyleam committed Mar 6, 2019
    This is consistent with the "already exists" branch of the code, is
    useful for debugging, and means we get an "assert clean" check before
    the push.
    Though none of that matters too much.  The local shell bits of
    datalad-pair's prepare_remote() exist at this point for local testing
    and will be unnecessary once DataLad's create-sibling supports setting
    up a local sibling.
Commits on Mar 5, 2019
  1. ENH: orchestrators: Log/fetch metadata of non-zero exits

    kyleam committed Mar 5, 2019
    When a job has a non-zero exit code, let the user know where to look
    for more information (getting the metadata files if needed).
    Closes #375.
  2. ENH: orchestrators: Configure .reproman/jobs/.gitignore

    kyleam committed Mar 5, 2019
    Ignore the submitter log file, which is currently used only by Condor.
    Ideally we would track this file, but we can't simply 'datalad add' it
    because that add call happens _within_ the job script.  This means the
    log file content can still change, leading to a bad annex file and
    failing 'git annex get' calls.
  3. ENH: orchestrators: Configure .reproman/jobs/.gitattributes

    kyleam committed Mar 5, 2019
    When we run the job on the remote dataset, the job template 'git add's
    status and idmap so that they are tracked in git.  But the local
    dataset for 'datalad run'-based orchestrators inconsistently puts
    these files under annex because the 'datalad add' call is done on the
    local machine.
    To consistently track these files in git, configure .gitattributes
    under .reproman/jobs rather than using 'git add' in the job template.
  4. ENH: orchestrators: Mark resurrections explicitly

    kyleam committed Mar 5, 2019
    Orchestrators are made to be resurrected, as does.  Add a
    parameter to Orchestrators.__init__ so that we can more reliably know
    if an orchestrator has been resurrected, which is needed to perform
    conditional setup and additional checks.
  5. ENH: plain orchestrator: Fetch some job metadata files

    kyleam committed Mar 5, 2019
    All of the other orchestrators make the metadata files available in
    the local dataset.  Update the plain orchestrator to download the ones
    that would most likely be useful to users, particularly for debugging.
  6. DOC: run: Remove meta_directory from listed job parameters

    kyleam committed Mar 5, 2019
    We hard code this as a subdirectory of working_directory and don't
    expose it via the job spec.  Let's not change that until there is a
    strong case made for doing so because knowing that the meta_directory
    is a subdirectory of working_directory allows us to reuse the
    "working_directory -> local_directory" mapping.
  7. RF: Prefer DataLad's .is_dirty() over .dirty

    kyleam committed Mar 5, 2019
    'git status' and GitRepo.is_dirty() return False when a directory
    contains no files, only other directories.  As of Datalad's
    e9be416f3 (2019-03-02), GitRepo.dirty now reports True.
  8. CLN+TST: Remove a redundant @mark.skipif_no_datalad

    kyleam committed Mar 5, 2019
    This will already be skipped due to the dataset fixture.
  9. TST: Improve readability of pycache_source test

    kyleam committed Mar 5, 2019
    The IDs of the parametrize test cases are in a separate list from the
    values.  Given the number of test cases, these IDs aren't very helpful
    as descriptions to a reader because it is hard to match an ID to a
    test case.  Store the IDs alongside the test values to make this
  10. ENH: pycache_source: Consider pyo files too

    kyleam committed Mar 5, 2019
    Update pycache_source() to handle files created under 'python -O ...'.
    Re: #378 (review)
Commits on Mar 4, 2019
  1. RF: orchestrators: Change a logged error to an exception

    kyleam committed Mar 4, 2019
    For a datalad-run fetch, if the expected outputs tar.gz doesn't exist,
    we log it as an error and return None.  Raise an exception instead
    because it is easier for Python callers to handle and is more
    consistent with what is done elsewhere in the orchestrators.
You can’t perform that action at this time.