Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make progress bar "backend" configurable, use the "log" one during tests #3396

Merged
merged 14 commits into from May 13, 2019

Conversation

@yarikoptic
Copy link
Member

commented May 7, 2019

Closes #3393

Adds LogProgressBar which logs completion (possibly partial) for the item.

Here are examples on how reporting looks like with `log` with some being interrupted or completed fully
$> sudo rm -rf QA simon ds00* face_place dartmouth-phantoms; DATALAD_UI_PROGRESSBAR=log datalad install -g ///openfmri/ds000105
[INFO   ] Cloning http://datasets.datalad.org/openfmri/ds000105 [1 other candidates] into '/tmp/ds000105' 
[INFO   ]  Cloning (counting objects) 100 objects done in 0 sec at 1880853 objects/sec 
[INFO   ]  Cloning (compressing objects) 68466 objects done in 0.001 sec at 52508176 objects/sec 
[INFO   ]  Cloning (receiving objects) 152614 objects done in 6 seconds at 25248 objects/sec 
[INFO   ]  Cloning (resolving stuff) 80063 objects done in 0.003 sec at 23785845 objects/sec 
[INFO   ] access to dataset sibling "datalad" not auto-enabled, enable with:
| 		datalad siblings -d "/tmp/ds000105" enable -s datalad 
install(ok): /tmp/ds000105 (dataset)
[INFO   ]  derivatives/mriqc/anatomical_sub-3.pdf 63.2 kB done in 0.172 sec at 366.9 kB/sec 
[INFO   ]  derivatives/mriqc/anatomical_sub-4.pdf 63.5 kB done in 0.13 sec at 490.0 kB/sec 
[INFO   ]  derivatives/mriqc/functional_group.pdf 182.8 kB done in 0.045 sec at 4.1 MB/sec 
[INFO   ]  derivatives/mriqc/funcMRIQC.csv 29.0 kB done in 0.198 sec at 146.6 kB/sec 
^C[WARNING] Still have 4 active progress bars when stopping 
[INFO   ]  Total partially (1.38% of 1.9 GB) done in 17 seconds at 1.5 MB/sec 
[INFO   ]  sub-1/anat/sub-1_T1w.nii.gz partially (84.71% of 6.0 MB) done in 15 seconds at 319.8 kB/sec 
[INFO   ]  sub-1/func/sub-1_task-objectviewing_run-02_bold.nii.gz partially (22.58% of 25.8 MB) done in 15 seconds at 373.5 kB/sec 
[INFO   ]  sub-1/func/sub-1_task-objectviewing_run-01_bold.nii.gz partially (28.36% of 25.7 MB) done in 15 seconds at 478.3 kB/sec 
[INFO   ]  sub-1/func/sub-1_task-objectviewing_run-03_bold.nii.gz partially (24.89% of 25.8 MB) done in 14 seconds at 428.5 kB/sec 
ERROR: 
Interrupted by user while doing magic: KeyboardInterrupt() [cmd.py:_process_one_line:358]
DATALAD_UI_PROGRESSBAR=log datalad install -g ///openfmri/ds000105  6.27s user 2.08s system 24% cpu 34.728 total

$> sudo rm -rf QA simon dartmouth-phantoms velasco; DATALAD_UI_PROGRESSBAR=log datalad install -g ///labs/tarr/face_place      
[INFO   ] Cloning http://datasets.datalad.org/labs/tarr/face_place [1 other candidates] into '/tmp/face_place' 
[INFO   ]  Cloning (counting objects) 100 objects done in 0 sec at 2853268 objects/sec 
[INFO   ]  Cloning (compressing objects) 28649 objects done in 0.001 sec at 38002092 objects/sec 
[INFO   ]  Cloning (receiving objects) 31593 objects done in 1.418 sec at 22275 objects/sec 
[INFO   ]  Cloning (resolving stuff) 387 objects done in 0.003 sec at 137477 objects/sec 
install(ok): /tmp/face_place (dataset)
[INFO   ] To obtain some keys we need to fetch an archive of size 42.5 MB 
[INFO   ]  MD5E-s42453590--70418661c05ac77945454b58f7a02789.zip 42.5 MB done in 32 seconds at 1.3 MB/sec 
[INFO   ] To obtain some keys we need to fetch an archive of size 28.8 MB 
^C[WARNING] Still have 1 active progress bars when stopping 
[INFO   ]  Total possibly partially done in 46 seconds at 1.3 MB/sec 
[INFO   ]  MD5E-s28831000--2d7e46964c86637e36c26745b8455aea.zip partially (28.79% of 28.8 MB) done in 6 seconds at 1.2 MB/sec 
ERROR: 
Interrupted by user while doing magic: KeyboardInterrupt() [cmd.py:_process_one_line:358]
DATALAD_UI_PROGRESSBAR=log datalad install -g ///labs/tarr/face_place  10.14s user 2.26s system 20% cpu 1:00.97 total

$> sudo rm -rf QA simon dartmouth-phantoms velasco; DATALAD_UI_PROGRESSBAR=log datalad install -g ///labs/tarr/face_place
[INFO   ] To obtain some keys we need to fetch an archive of size 28.8 MB 
[INFO   ]  MD5E-s28831000--2d7e46964c86637e36c26745b8455aea.zip 28.8 MB done in 14 seconds at 2.0 MB/sec 
[INFO   ] To obtain some keys we need to fetch an archive of size 118.2 MB 
[INFO   ]  MD5E-s118229974--970c5c314ec41a2950e98aeefb2faf44.zip 118.2 MB done in a minute at 1.1 MB/sec 
[INFO   ] To obtain some keys we need to fetch an archive of size 16.7 MB 
[INFO   ]  MD5E-s16663731--4735c5882fd6736c1b0829c428a40503.zip 16.7 MB done in 11 seconds at 1.4 MB/sec 
[INFO   ] To obtain some keys we need to fetch an archive of size 16.3 MB 
[INFO   ]  MD5E-s16251761--d049c7c7316b38e847833f574a1ba2ad.zip 16.3 MB done in 13 seconds at 1.2 MB/sec 
[INFO   ]  Total 40.8 MB done in 2 minutes at 233.6 kB/sec 
action summary:
  get (ok: 5290)
  install (notneeded: 1)
DATALAD_UI_PROGRESSBAR=log datalad install -g ///labs/tarr/face_place  69.89s user 21.58s system 50% cpu 3:02.77 total

Ideally should have dedicated test(s) but not sure if I would have juice for that ATM, but let's see if some bugs get revealed by just using it through the tests

yarikoptic added some commits May 7, 2019

BF: apply OP.MASK for git clone operation before considering
Otherwise we were getting progress bars recreated upon some composite
op_code (e.g. 32+2 = 34) being received, which we later anyways would
& with the mask to get the code we know how to handle.  Most likely
there is more to it, but at least this would make it more consistent
and avoid duplicate/confusing log entries with upcoming log based
progress reporting
ENH+RF: LogProgressBar to log upon .finish()
Progress bar might be finished incomplete, then more details about progress is output.

Also now propagate common progress bar options such as label, unit in a (more) consistent
signature

@yarikoptic yarikoptic changed the base branch from master to 0.11.x May 7, 2019

@yarikoptic

This comment has been minimized.

Copy link
Member Author

commented May 7, 2019

@effigies how does this one look to you? ;)

@effigies

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

Definitely much cleaner. And it works without any obvious bugs on my side.

@codecov

This comment has been minimized.

Copy link

commented May 8, 2019

Codecov Report

Merging #3396 into 0.11.x will decrease coverage by <.01%.
The diff coverage is 83.75%.

Impacted file tree graph

@@            Coverage Diff            @@
##           0.11.x   #3396      +/-   ##
=========================================
- Coverage      91%     91%   -0.01%     
=========================================
  Files         254     254              
  Lines       33323   33385      +62     
=========================================
+ Hits        30327   30383      +56     
- Misses       2996    3002       +6
Impacted Files Coverage Δ
datalad/log.py 89.9% <ø> (ø) ⬆️
datalad/interface/common_cfg.py 100% <ø> (ø) ⬆️
datalad/ui/__init__.py 97.67% <ø> (-0.06%) ⬇️
datalad/ui/tests/test_dialog.py 98.9% <100%> (+0.05%) ⬆️
datalad/support/gitrepo.py 89.13% <100%> (+0.02%) ⬆️
datalad/support/tests/test_annexrepo.py 96.53% <100%> (ø) ⬆️
datalad/ui/progressbars.py 83.1% <79.24%> (-3.04%) ⬇️
datalad/support/annexrepo.py 88.98% <87.5%> (+0.01%) ⬆️
datalad/ui/dialog.py 92.35% <88.88%> (-0.86%) ⬇️
datalad/downloaders/http.py 85.31% <0%> (+2.77%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b368349...11a6408. Read the comment docs.

@codecov

This comment has been minimized.

Copy link

commented May 8, 2019

Codecov Report

Merging #3396 into 0.11.x will decrease coverage by 48.42%.
The diff coverage is 69.11%.

Impacted file tree graph

@@             Coverage Diff             @@
##           0.11.x    #3396       +/-   ##
===========================================
- Coverage      91%   42.58%   -48.43%     
===========================================
  Files         254      254               
  Lines       33323    33362       +39     
===========================================
- Hits        30327    14207    -16120     
- Misses       2996    19155    +16159
Impacted Files Coverage Δ
datalad/ui/__init__.py 83.72% <ø> (-14.01%) ⬇️
datalad/interface/common_cfg.py 100% <ø> (ø) ⬆️
datalad/support/gitrepo.py 60.5% <100%> (-28.61%) ⬇️
datalad/ui/dialog.py 41.76% <55.55%> (-51.45%) ⬇️
datalad/support/annexrepo.py 44.04% <62.5%> (-44.93%) ⬇️
datalad/ui/progressbars.py 51.38% <71.42%> (-34.75%) ⬇️
datalad/support/tests/utils.py 13.04% <0%> (-86.96%) ⬇️
datalad/support/tests/test_stats.py 13.11% <0%> (-86.89%) ⬇️
datalad/support/tests/test_repodates.py 13.46% <0%> (-86.54%) ⬇️
datalad/cmdline/tests/test_formatters.py 14.28% <0%> (-85.72%) ⬇️
... and 213 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b368349...d28d6a6. Read the comment docs.

BF: do not bother estimating /sec if 0 sec passed
Apparently on windows time.time is likely to return full seconds, causing
test(s) to fail

See https://ci.appveyor.com/project/mih/datalad/builds/24380171/job/6sfm21kbsaajgv9y
@kyleam
Copy link
Member

left a comment

Thanks. I think that it's good we're exposing a configuration option for this (discussed a bit in gh-3092 and 51cc0b0).

Overall it looks good to me. Some minor general comments:

  • I was surprised this series made no mention of log_progress. LogProgressBar and log.log_progress do related but different things. It'd be nice if the docstring of LogProgressBar pointed to log.log_progress as a way to get progress bars when attached to a tty but incremental log messages otherwise (as opposed to just the final log message here). And log_progress could point to LogProgressBar as well.

  • Output like "... 10 items done in 10 seconds at 0.9984752098526601 items/sec" is hyper-precise.

'datalad.ui.progressbar': {
'ui': ('question', {
'title': 'UI progress bars',
'text': 'Which backend for progress reporting to use unless specified otherwise?'}),

This comment has been minimized.

Copy link
@kyleam

kyleam May 8, 2019

Member

Perhaps "Default backend for progress reporting"?

def __init__(self, *args, **kwargs):
super(LogProgressBar, self).__init__(*args, **kwargs)
# I think we never generate progress bars unless we are the beginning
# of reporting something lengthy. .start is not always invoked

This comment has been minimized.

Copy link
@kyleam

kyleam May 8, 2019

Member

s/are/are at/, though I'm missing the connection between .start being invoked and recording the start time here.

else:
# well well -- we still probably have some issue with
# over-reporting when getting data from datalad-archives
# Instead of providing non-sense % here, j

This comment has been minimized.

Copy link
@kyleam

kyleam May 8, 2019

Member

Incomplete comment?

@yarikoptic yarikoptic added this to the 0.11.x milestone May 10, 2019

@yarikoptic

This comment has been minimized.

Copy link
Member Author

commented May 10, 2019

I was surprised this series made no mention of log_progress. LogProgressBar and log.log_progress do related but different things. It'd be nice if the docstring of LogProgressBar pointed to log.log_progress as a way to get progress bars when attached to a tty but incremental log messages otherwise (as opposed to just the final log message here). And log_progress could point to LogProgressBar as well.

doh -- forgot about it! will do cross reference. Ideally we should (not in this PR hopefully) may be unify the msg format if applicable?

@kyleam

This comment has been minimized.

Copy link
Member

commented May 10, 2019

yarikoptic added some commits May 10, 2019

RF: use custom _naturalfloat for the best presentation of numbers
I found no other alternative, with this one output would look like

[INFO   ] Cloning http://datasets.datalad.org/labs/tarr/face_place [1 other candidates] into '/tmp/face_place'
[INFO   ]  Cloning (counting objects) 100 objects done in 6.69956e-05 sec at 1492635 objects/sec
[INFO   ]  Cloning (compressing objects) 28649 objects done in 0.00248098 sec at 11547436 objects/sec
[INFO   ]  Cloning (receiving objects) 31593 objects done in 1.61515 sec at 19560 objects/sec
[INFO   ]  Cloning (resolving stuff) 387 objects done in 0.00757694 sec at 51076 objects/sec
install(ok): /tmp/face_place (dataset)
[INFO   ] To obtain some keys we need to fetch an archive of size 42.5 MB
[INFO   ]  MD5E-s42453590--70418661c05ac77945454b58f7a02789.zip 42.5 MB done in 32 seconds at 1.3 MB/sec
[INFO   ] To obtain some keys we need to fetch an archive of size 28.8 MB
[INFO   ]  MD5E-s28831000--2d7e46964c86637e36c26745b8455aea.zip 28.8 MB done in 23 seconds at 1.2 MB/sec
[INFO   ] To obtain some keys we need to fetch an archive of size 118.2 MB
[INFO   ]  MD5E-s118229974--970c5c314ec41a2950e98aeefb2faf44.zip 118.2 MB done in a minute at 1.6 MB/sec
[INFO   ] To obtain some keys we need to fetch an archive of size 16.7 MB
[INFO   ]  MD5E-s16663731--4735c5882fd6736c1b0829c428a40503.zip 16.7 MB done in 12 seconds at 1.3 MB/sec
[INFO   ] To obtain some keys we need to fetch an archive of size 16.3 MB
[INFO   ]  MD5E-s16251761--d049c7c7316b38e847833f574a1ba2ad.zip 16.3 MB done in 11 seconds at 1.4 MB/sec
[INFO   ]  Total 50.3 MB done in 4 minutes at 208.5 kB/sec
@yarikoptic

This comment has been minimized.

Copy link
Member Author

commented May 10, 2019

ah ok, thanks for the clarification. I have pushed my changes to address your comments -- thank you for the review! so now it would look like

[INFO   ] Cloning http://datasets.datalad.org/labs/tarr/face_place [1 other candidates] into '/tmp/face_place' 
[INFO   ]  Cloning (counting objects) 100 objects done in 5.79357e-05 sec at 1726051 objects/sec 
[INFO   ]  Cloning (compressing objects) 28649 objects done in 0.00212502 sec at 13481725 objects/sec 
[INFO   ]  Cloning (receiving objects) 31593 objects done in 1.68729 sec at 18724 objects/sec 
[INFO   ]  Cloning (resolving stuff) 387 objects done in 0.00789118 sec at 49042 objects/sec 
install(ok): /tmp/face_place (dataset)
[INFO   ] To obtain some keys we need to fetch an archive of size 42.5 MB 
[INFO   ]  MD5E-s42453590--70418661c05ac77945454b58f7a02789.zip 42.5 MB done in 35 seconds at 1.2 MB/sec 
[INFO   ] To obtain some keys we need to fetch an archive of size 28.8 MB 
...

still could be considered a bit "too precise" but IMHO not too bad ;) improve to your liking

@kyleam

This comment has been minimized.

Copy link
Member

commented May 10, 2019

Thanks for the updates. I reworded one of the cross-references. Feel free to drop that commit if it doesn't look ok to you.

still could be considered a bit "too precise" but IMHO not too bad ;) improve to your liking

I'd still lop off some digits, but that's both subjective and doesn't matter :]

@kyleam

kyleam approved these changes May 10, 2019

@kyleam kyleam merged commit 11a6408 into datalad:0.11.x May 13, 2019

3 of 5 checks passed

codecov/patch 83.75% of diff hit (target 91%)
Details
codecov/project 91% (-0.01%) compared to b368349
Details
WIP ready for review
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

kyleam added a commit that referenced this pull request May 13, 2019

kyleam added a commit that referenced this pull request May 13, 2019

yarikoptic added a commit to yarikoptic/datalad that referenced this pull request May 14, 2019

Merge branch '0.11.x' into bf-wrapt-troubleshoot
* 0.11.x: (615 commits)
  CHANGELOG.md: Update for dataladgh-3396
  DOC: log_progress: Reword cross-reference from 96b45b4
  ENH: adjust code comments
  ENH: adjust common_cfg description for datalad.ui.progressbar
  RF: use custom _naturalfloat for the best presentation of numbers
  ENH(DOC): cross reference LogProgressBar and log_progress in docstrings
  DOC: log_progress: Correct parameter name in docstring
  RF: versioned depends on sphinx within setup.py itself
  BF: do not bother estimating /sec if 0 sec passed
  RF/BF: reorder ProgressBarBase kwargs to match the one of the child
  BF(TST): making tests account for the changes
  NF: QuietConsoleLog to be used during tests (instead of SilentConsoleLog with no progress whatsoever)
  RF: AnnexRepo report to progress bar(s) that finish is partial if interrupted
  ENH+RF: LogProgressBar to log upon .finish()
  BF: report that gitrepo clone progress is about objects (not default to tqdm progress bar Bytes)
  ENH: datalad.ui.progressbar configuration
  BF: apply OP.MASK for git clone operation before considering
  CHANGELOG.md: Touch up 0.11.5 entries
  CHANGELOG.md: Second batch for 0.11.5
  RF: annexrepo: Tighten decision to commit with temporary index
  ...

yarikoptic added a commit to yarikoptic/datalad that referenced this pull request May 21, 2019

Merge remote-tracking branch 'origin/0.11.x' into nf-zenodo
* origin/0.11.x: (3256 commits)
  MNT: Avoid invalid escape sequences in strings
  BF: export_to_figshare: Don't test identity of string literal
  BF(TST): do not assume user naiveness - treat any url-like looking path as a path
  BF: Check for /, \ or # in the username@hostname part while detecting SSHRI
  CHANGELOG.md: Add entry for dataladgh-3374
  BF: revert back (remove) check for path being PathRI
  BF: list annex-transfer also in cmdline opt choice for "what"
  RF: convert imports into "tupled lists" of what to import
  BF: stop relying on setup.py runtime environment using platform.dist
  CHANGELOG.md: Update for dataladgh-3407
  RF: Set precedence to datalad.ui.color, NO_COLOR, datalad.ui.ui.is_interactive
  TEST: Test ANSI colors tools directly
  ENH: ls: Replace custom logic with color_word()
  RF: Move datalad.ui.color to common_cfg
  DOC: Add Zenodo
  NF: Add check for whether color is enabled
  RF: disable wrapt workaround
  CHANGELOG.md: Update for dataladgh-3396
  DOC: log_progress: Reword cross-reference from 96b45b4
  ENH: adjust code comments
  ...

yarikoptic added a commit to yarikoptic/datalad that referenced this pull request May 21, 2019

Merge remote-tracking branch 'origin/0.11.x' into nf-zenodo
* origin/0.11.x: (3256 commits)
  MNT: Avoid invalid escape sequences in strings
  BF: export_to_figshare: Don't test identity of string literal
  BF(TST): do not assume user naiveness - treat any url-like looking path as a path
  BF: Check for /, \ or # in the username@hostname part while detecting SSHRI
  CHANGELOG.md: Add entry for dataladgh-3374
  BF: revert back (remove) check for path being PathRI
  BF: list annex-transfer also in cmdline opt choice for "what"
  RF: convert imports into "tupled lists" of what to import
  BF: stop relying on setup.py runtime environment using platform.dist
  CHANGELOG.md: Update for dataladgh-3407
  RF: Set precedence to datalad.ui.color, NO_COLOR, datalad.ui.ui.is_interactive
  TEST: Test ANSI colors tools directly
  ENH: ls: Replace custom logic with color_word()
  RF: Move datalad.ui.color to common_cfg
  DOC: Add Zenodo
  NF: Add check for whether color is enabled
  RF: disable wrapt workaround
  CHANGELOG.md: Update for dataladgh-3396
  DOC: log_progress: Reword cross-reference from 96b45b4
  ENH: adjust code comments
  ...

yarikoptic added a commit that referenced this pull request May 28, 2019

Merge tag '0.11.5' into debian
0.11.5 (May 23, 2019) -- stability is not overrated

Should be faster and less buggy, with a few enhancements.

 Fixes

- [create-sibling][]  ([#3318][])
  - Siblings are no longer configured with a post-update hook unless a
    web interface is requested with `--ui`.
  - `git submodule update --init` is no longer called from the
    post-update hook.
  - If `--inherit` is given for a dataset without a superdataset, a
    warning is now given instead of raising an error.
- The internal command runner failed on Python 2 when its `env`
  argument had unicode values.  ([#3332][])
- The safeguard that prevents creating a dataset in a subdirectory
  that already contains tracked files for another repository failed on
  Git versions before 2.14.  For older Git versions, we now warn the
  caller that the safeguard is not active.  ([#3347][])
- A regression introduced in v0.11.1 prevented [save][] from committing
  changes under a subdirectory when the subdirectory was specified as
  a path argument.  ([#3106][])
- A workaround introduced in v0.11.1 made it possible for [save][] to
  do a partial commit with an annex file that has gone below the
  `annex.largefiles` threshold.  The logic of this workaround was
  faulty, leading to files being displayed as typechanged in the index
  following the commit.  ([#3365][])
- The resolve_path() helper confused paths that had a semicolon for
  SSH RIs.  ([#3425][])
- The detection of SSH RIs has been improved.  ([#3425][])

 Enhancements and new features

- The internal command runner was too aggressive in its decision to
  sleep.  ([#3322][])
- The "INFO" label in log messages now retains the default text color
  for the terminal rather than using white, which only worked well for
  terminals with dark backgrounds.  ([#3334][])
- A short flag `-R` is now available for the `--recursion-limit` flag,
  a flag shared by several subcommands.  ([#3340][])
- The authentication logic for [create-sibling-github][] has been
  revamped and now supports 2FA.  ([#3180][])
- New configuration option `datalad.ui.progressbar` can be used to
  configure the default backend for progress reporting ("none", for
  example, results in no progress bars being shown).  ([#3396][])
- A new progress backend, available by setting datalad.ui.progressbar
  to "log", replaces progress bars with a log message upon completion
  of an action.  ([#3396][])
- DataLad learned to consult the [NO_COLOR][] environment variable and
  the new `datalad.ui.color` configuration option when deciding to
  color output.  The default value, "auto", retains the current
  behavior of coloring output if attached to a TTY ([#3407][]).
- [clean][] now removes annex transfer directories, which is useful
  for cleaning up failed downloads. ([#3374][])
- [clone][] no longer refuses to clone into a local path that looks
  like a URL, making its behavior consistent with `git clone`.
  ([#3425][])
- [wtf][]
  - Learned to fall back to the `dist` package if `platform.dist`,
    which has been removed in the yet-to-be-release Python 3.8, does
    not exist.  ([#3439][])
  - Gained a `--section` option for limiting the output to specific
    sections and a `--decor` option, which currently knows how to
    format the output as GitHub's `<details>` section.  ([#3440][])

* tag '0.11.5': (96 commits)
  [DATALAD RUNCMD] make update-changelog
  Version boost and finalize CHANGELOG.md record
  ENH: new Makefile rule linkissues-changelog to link issues, which now will also be prerequisite for update-changelog
  CHANGELOG.md: Add entries for recently merged PRs
  ENH: require "distro" for python >= 3.8
  ENH: compat with python 3.8 which removed .dist -- try distro
  CLN: wtf: Remove unused (and duplicated) import
  DOC: wtf: Avoid double period in -S's description
  ENH: -D|--decor html_details -- to make it ready for pasting to github issue without clutter
  BF: assure bytes while giving to pyperclip upon its demand (on Py2)
  RF: move always present path + type "section" into "location" section, retain order of sections from cmdline
  RF: switch from nargs="*" to action=append for wtf -S
  ENH: wtf -S to specify which sections to query/display (by default -- all)
  MNT: Avoid invalid escape sequences in strings
  BF: export_to_figshare: Don't test identity of string literal
  BF(TST): do not assume user naiveness - treat any url-like looking path as a path
  BF: Check for /, \ or # in the username@hostname part while detecting SSHRI
  CHANGELOG.md: Add entry for gh-3374
  BF: revert back (remove) check for path being PathRI
  BF: list annex-transfer also in cmdline opt choice for "what"
  ...

yarikoptic added a commit that referenced this pull request May 28, 2019

Merge tag '0.11.5' into debian -- got two more minor changes
0.11.5 (May 23, 2019) -- stability is not overrated

Should be faster and less buggy, with a few enhancements.

 Fixes

- [create-sibling][]  ([#3318][])
  - Siblings are no longer configured with a post-update hook unless a
    web interface is requested with `--ui`.
  - `git submodule update --init` is no longer called from the
    post-update hook.
  - If `--inherit` is given for a dataset without a superdataset, a
    warning is now given instead of raising an error.
- The internal command runner failed on Python 2 when its `env`
  argument had unicode values.  ([#3332][])
- The safeguard that prevents creating a dataset in a subdirectory
  that already contains tracked files for another repository failed on
  Git versions before 2.14.  For older Git versions, we now warn the
  caller that the safeguard is not active.  ([#3347][])
- A regression introduced in v0.11.1 prevented [save][] from committing
  changes under a subdirectory when the subdirectory was specified as
  a path argument.  ([#3106][])
- A workaround introduced in v0.11.1 made it possible for [save][] to
  do a partial commit with an annex file that has gone below the
  `annex.largefiles` threshold.  The logic of this workaround was
  faulty, leading to files being displayed as typechanged in the index
  following the commit.  ([#3365][])
- The resolve_path() helper confused paths that had a semicolon for
  SSH RIs.  ([#3425][])
- The detection of SSH RIs has been improved.  ([#3425][])

 Enhancements and new features

- The internal command runner was too aggressive in its decision to
  sleep.  ([#3322][])
- The "INFO" label in log messages now retains the default text color
  for the terminal rather than using white, which only worked well for
  terminals with dark backgrounds.  ([#3334][])
- A short flag `-R` is now available for the `--recursion-limit` flag,
  a flag shared by several subcommands.  ([#3340][])
- The authentication logic for [create-sibling-github][] has been
  revamped and now supports 2FA.  ([#3180][])
- New configuration option `datalad.ui.progressbar` can be used to
  configure the default backend for progress reporting ("none", for
  example, results in no progress bars being shown).  ([#3396][])
- A new progress backend, available by setting datalad.ui.progressbar
  to "log", replaces progress bars with a log message upon completion
  of an action.  ([#3396][])
- DataLad learned to consult the [NO_COLOR][] environment variable and
  the new `datalad.ui.color` configuration option when deciding to
  color output.  The default value, "auto", retains the current
  behavior of coloring output if attached to a TTY ([#3407][]).
- [clean][] now removes annex transfer directories, which is useful
  for cleaning up failed downloads. ([#3374][])
- [clone][] no longer refuses to clone into a local path that looks
  like a URL, making its behavior consistent with `git clone`.
  ([#3425][])
- [wtf][]
  - Learned to fall back to the `dist` package if `platform.dist`,
    which has been removed in the yet-to-be-release Python 3.8, does
    not exist.  ([#3439][])
  - Gained a `--section` option for limiting the output to specific
    sections and a `--decor` option, which currently knows how to
    format the output as GitHub's `<details>` section.  ([#3440][])

* tag '0.11.5':
  BF: make test for url download more reliable in cases where connection fails
  RF: remove stale commented out duecredit in setup.py.  It has now its own section

@yarikoptic yarikoptic deleted the yarikoptic:enh-pbars branch Jun 3, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.