op.realpath() -> Path.resolve() #4057

mih · 2020-01-20T11:23:28Z

This aims to fix #4055 and fix #4192

make with_tempfile report properly resolved longpaths on windows to satisfy all those tests that perform lexical comparisons of paths
monkey-patch Path.resolve() to support strict=False on PY35
watch tests and observe making patching to work on PY35
implement full replacement in all parts that could potentially impact any path comparison

codecov · 2020-01-20T12:15:35Z

Codecov Report

Merging #4057 into master will decrease coverage by 0.16%.
The diff coverage is 90.75%.

@@            Coverage Diff             @@
##           master    #4057      +/-   ##
==========================================
- Coverage   88.74%   88.58%   -0.17%     
==========================================
  Files         282      282              
  Lines       36888    37034     +146     
==========================================
+ Hits        32738    32807      +69     
- Misses       4150     4227      +77

Impacted Files	Coverage Δ
datalad/support/tests/test_network.py	`100% <ø> (ø)`	⬆️
datalad/cmdline/tests/test_helpers.py	`100% <100%> (ø)`	⬆️
datalad/support/tests/test_gitrepo.py	`99.75% <100%> (-0.01%)`	⬇️
datalad/tests/test_utils.py	`95.96% <100%> (ø)`	⬆️
datalad/support/tests/test_annexrepo.py	`95.27% <100%> (ø)`	⬆️
datalad/distribution/tests/test_uninstall.py	`100% <100%> (ø)`	⬆️
datalad/support/archives.py	`89.47% <100%> (ø)`	⬆️
datalad/support/repo.py	`95.74% <100%> (ø)`	⬆️
datalad/support/gitrepo.py	`90.68% <100%> (ø)`	⬆️
datalad/interface/tests/test_rerun.py	`100% <100%> (ø)`	⬆️
... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0741da6...5b9082a. Read the comment docs.

mih · 2020-01-20T16:05:57Z

Windows test failures are funky, and do not replicate on a real win10 box or appveyor.... 🙄

datalad/utils.py

datalad/distribution/dataset.py

datalad/support/annexrepo.py

datalad/support/gitrepo.py

mih · 2020-02-25T09:43:25Z

Force-pushed a rebased PR, with all discussed issues addressed. Let's see....

mih · 2020-02-25T12:59:26Z

Grrr, sank another day into this one. If you see the issue(s), please HELP!!! Thx!

This one approach to a clean solution to dataladgh-4055 without having to introduce a custom helper function.

Importantly, this makes the with_tempfile decorator use this new approach, which provides any test with proper longpaths that afford lexical comparisions on Windows.

…code Raise if an archive path outside a repository was given to add_archive_content.This scenario is not supported.

Must be more gentle, cannot have really obscure file names on windows (limit similar to other places in the code).

Not that it would matter other than making the test fail.

mih · 2020-03-08T17:41:29Z

Still not worky worky :(

…ss only

mih · 2020-03-09T07:30:11Z

The remaining test failures with automagic IO only occur with PY36+ and git-annex-standalone=7.20190819+git2-g908476a9b-1~ndall+1 , but not, e.g., with git-annex=7.20191230.
Fun times....

mih · 2020-03-09T10:20:02Z

Once this PR is merged, we can finally start hunting down Github CI issues on windows (those that remain, and are not due to invalid lexical short path comparisons)

rmtree(): device or resource busy and friends...

mih · 2020-03-09T12:34:44Z

This is done.

mih mentioned this pull request Jan 20, 2020

BF: Make all known_failure GitRepo tests run on windows #4054

Closed

1 task

mih force-pushed the bf-4055 branch from d268225 to 27f116a Compare Jan 21, 2020

bpoldrack reviewed Jan 23, 2020

View changes

datalad/utils.py Show resolved Hide resolved

bpoldrack reviewed Jan 23, 2020

View changes

bpoldrack mentioned this pull request Jan 29, 2020

NF: clone: --reckless='ephemeral' #4099

Merged

bpoldrack mentioned this pull request Feb 11, 2020

reckless=ephemeral test fails on windows #4131

Closed

yarikoptic added this to the 0.12.3 milestone Feb 20, 2020

mih mentioned this pull request Feb 25, 2020

test_get_open_files() fails on Windows #4192

Closed

mih force-pushed the bf-4055 branch from 27f116a to 3a59fd1 Compare Feb 25, 2020

mih force-pushed the bf-4055 branch from e6a61ec to 795d556 Compare Feb 25, 2020

yarikoptic modified the milestones: 0.12.3, 0.12.x Feb 27, 2020

mih force-pushed the bf-4055 branch from 795d556 to 12445d1 Compare Mar 8, 2020

mih added 8 commits Mar 8, 2020

RF: Monkey-patch Path.resolve() to bring strict=False to PY35

3673fca

This one approach to a clean solution to dataladgh-4055 without having to introduce a custom helper function.

RF: All Path imports from datalad.utils

f00375a

RF: Replace op.realpath with Path.resolve()

8a8b0fc

Importantly, this makes the with_tempfile decorator use this new approach, which provides any test with proper longpaths that afford lexical comparisions on Windows.

RF: Simplify and explain equality test for Datasets

2bb9fcc

BF: Robustify Dataset equality test for non-existing Datasets on Windows

669e273

BF: Do not rely on implicit/undocumented assumption about underlying …

59a8f82

…code Raise if an archive path outside a repository was given to add_archive_content.This scenario is not supported.

BF: Avoid butchered file:// URLs from Windows short paths

3391843

Must be more gentle, cannot have really obscure file names on windows (limit similar to other places in the code).

TST: Path.resolve() gives different capitalization on Windows

99af8a1

Not that it would matter other than making the test fail.

mih force-pushed the bf-4055 branch from 12445d1 to 99af8a1 Compare Mar 8, 2020

BF: Improper use of resolve() when we want to inspect a symlink addre…

c2cd9e4

…ss only

BF: Reimplement test to avoid functions patched by automagic IO

633261e

TST: Avoid leaving stuff behind that can prevent a run on NFS to success

5b9082a

rmtree(): device or resource busy and friends...

mih merged commit 0b2e20e into datalad:master Mar 9, 2020
12 of 18 checks passed

mih deleted the bf-4055 branch Mar 9, 2020

op.realpath() -> Path.resolve() #4057

op.realpath() -> Path.resolve() #4057

mih commented Jan 20, 2020 •

edited

codecov bot commented Jan 20, 2020 •

edited

mih commented Jan 20, 2020

mih commented Feb 25, 2020

mih commented Feb 25, 2020

mih commented Mar 8, 2020

mih commented Mar 9, 2020 •

edited

mih commented Mar 9, 2020

mih commented Mar 9, 2020

op.realpath() -> Path.resolve() #4057

op.realpath() -> Path.resolve() #4057

Conversation

mih commented Jan 20, 2020 • edited

codecov bot commented Jan 20, 2020 • edited

Codecov Report

mih commented Jan 20, 2020

mih commented Feb 25, 2020

mih commented Feb 25, 2020

mih commented Mar 8, 2020

mih commented Mar 9, 2020 • edited

mih commented Mar 9, 2020

mih commented Mar 9, 2020

mih commented Jan 20, 2020 •

edited

codecov bot commented Jan 20, 2020 •

edited

mih commented Mar 9, 2020 •

edited