Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NF: rerun: Add --explicit flag #3498

Merged
merged 1 commit into from Jul 4, 2019
Merged

Conversation

@kyleam
Copy link
Contributor

@kyleam kyleam commented Jun 28, 2019

Closes #3074.

@yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Jul 1, 2019

Thanks again! I guess this test should be skipped in direct mode?

======================================================================
ERROR: datalad.interface.tests.test_run.test_rerun_explicit
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/datalad/tests/utils.py", line 442, in newfunc
    return t(*(arg + (d,)), **kw)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/datalad/interface/tests/test_run.py", line 935, in test_rerun_explicit
    eq_(orig_head, ds.repo.get_hexsha("master~1"))
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/datalad/support/gitrepo.py", line 1457, in get_hexsha
    commitish)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/datalad/support/gitrepo.py", line 1431, in format_commit
    raise ValueError("Unknown commit identifier: %s" % commitish)
ValueError: Unknown commit identifier: master~1

@kyleam kyleam force-pushed the rerun-explicit branch from 34d5ace to edbdd45 Jul 1, 2019
@codecov
Copy link

@codecov codecov bot commented Jul 1, 2019

Codecov Report

No coverage uploaded for pull request base (0.11.x@9a65227). Click here to learn what that means.
The diff coverage is 13.33%.

Impacted file tree graph

@@            Coverage Diff            @@
##             0.11.x    #3498   +/-   ##
=========================================
  Coverage          ?   60.02%           
=========================================
  Files             ?      252           
  Lines             ?    34240           
  Branches          ?        0           
=========================================
  Hits              ?    20551           
  Misses            ?    13689           
  Partials          ?        0
Impacted Files Coverage Δ
datalad/interface/rerun.py 18.86% <50%> (ø)
datalad/interface/tests/test_run.py 40.44% <7.69%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a65227...34adf74. Read the comment docs.

@kyleam
Copy link
Contributor Author

@kyleam kyleam commented Jul 1, 2019

I guess this test should be skipped in direct mode?

Indeed. Pushed update.

run() will execute a command in a dirty working tree if --explicit is
passed, but rerun() requires a clean state.  Add an --explicit flag to
rerun() that makes all the underlying run calls operate with
explicit=True.

In some cases, this doesn't play well with --since (because it applies
to all commits, and they all may not have the appropriate outputs
specified) or with --onto because the Git operations we
perform (particularly checkout and cherry-pick) may fail due to
conflicts with the working tree modifications.

Instead of (or in addition to) adding --explicit to rerun, we could
store the explicit setting in the run record itself.  This information
isn't essential to the run, but it would allow us to selectively rerun
a subset of the commits in a range with explicit=True.  That's more
involved and intrusive, though, so let's hold off on that until there
is actually a demand for it.

Closes datalad#3074.
@yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Jul 4, 2019

thanks again -- will merge for real world testing ;)

@yarikoptic yarikoptic merged commit 8d8e41c into datalad:0.11.x Jul 4, 2019
2 of 3 checks passed
@kyleam kyleam deleted the rerun-explicit branch Jul 8, 2019
@yarikoptic yarikoptic added this to the Release 0.11.6 milestone Jul 26, 2019
yarikoptic added a commit that referenced this issue Jul 31, 2019
0.11.6 (Jul 30, 2019) -- am I the last of 0.11.x?

Primarily bug fixes to achieve more robust performance

Fixes

- Our tests needed various adjustments to keep up with upstream
  changes in Travis and Git. ([#3479][]) ([#3492][]) ([#3493][])

- `AnnexRepo.is_special_annex_remote` was too selective in what it
  considered to be a special remote.  ([#3499][])

- We now provide information about unexpected output when git-annex is
  called with `--json`.  ([#3516][])

- Exception logging in the `__del__` method of `GitRepo` and
  `AnnexRepo` no longer fails if the names it needs are no longer
  bound.  ([#3527][])

- [addurls][] botched the construction of subdataset paths that were
  more than two levels deep and failed to create datasets in a
  reliable, breadth-first order.  ([#3561][])

- Cloning a `type=git` special remote showed a spurious warning about
  the remote not being enabled.  ([#3547][])

Enhancements and new features

- For calls to git and git-annex, we disable automatic garbage
  collection due to past issues with GitPython's state becoming stale,
  but doing so results in a larger .git/objects/ directory that isn't
  cleaned up until garbage collection is triggered outside of DataLad.
  Tests with the latest GitPython didn't reveal any state issues, so
  we've re-enabled automatic garbage collection.  ([#3458][])

- [rerun][] learned an `--explicit` flag, which it relays to its calls
  to [run][[]].  This makes it possible to call `rerun` in a dirty
  working tree ([#3498][]).

- The [metadata][] command aborts earlier if a metadata extractor is
  unavailable.  ([#3525][])

* tag '0.11.6': (56 commits)
  [DATALAD RUNCMD] make update-changelog
  finalize CHANGELOG.md entry and boost version
  BF(DOC): close [create] with [] to not cause WARNING by md-strict pandoc
  CHANGELOG.md: Link entry from b3e8adb
  CHANGELOG.md: Add entry for gh-3547
  CHANGELOG.md: Add entry for gh-3561
  CHANGELOG.md: Add link for addurls
  RF: inform about special remotes based on autoenable config
  CHANGELOG.md: Second batch for 0.11.6
  BF: addurls: Process datasets in a stable, breadth-first order
  BF: addurls: Fix construction of nested subpaths
  TST: addurls: Don't hard-code path separator
  BF(TST): skip test_v7_detached_get in direct mode - fails to annex upgrade
  TST: benchmark-travis-pr: Swap 'pip install' and 'git show'
  TST: benchmark-travis-pr: Move repeated logic to run_asv()
  TST: benchmark-travis-pr: Support other bases
  TST: benchmark-travis-pr: Tweak message about current HEAD
  TST: benchmark-travis-pr: Simplify two git commands into one
  TST: benchmark-travis-pr: Reorder and break up lines
  TST: benchmark-travis-pr: Move command for running asv into function
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants