Display path relative to temporary repo when get/import-ing files #2798

skshetry · 2019-11-16T02:17:45Z

❗ Have you followed the guidelines in the Contributing to DVC list?
📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.
❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addresses. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Description

This PR fixes #2605. This will print the relative path of files being downloaded in the console.

tests/unit/output/test_local.py

efiop · 2019-11-16T09:35:04Z

Thanks for the PR @skshetry ! Just a heads up: some tests failed, please take a look at travis logs. 🙂

skshetry · 2019-11-16T09:51:54Z

tests/unit/output/test_local.py

+        o = self._get_output(wdir=work_dir)
+        self.assertEqual("path", str(o))
+
+    def test_str_if_workdir_outside_repo(self):


@efiop, this test is failing for SSH/HDFS/s3/GS as they are remote. Others are passing.
Should I just separate this test (and, keep it only for local)?

I have separated the test for now. Tell me, I feel, I have done something wrong here 😃

Now the three old tests are not tested for other remotes than local.

Considering all the test complications I suggest not touching existing ones, but adding a separate pytest-style test to check output stringification for an output in external repo.

Now the three old tests are not tested for other remotes than local.

They do run, as the above TestCase is not changed at all. Or, did you mean this specific example?

Considering all the test complications I suggest not touching existing ones, but adding a separate pytest-style test to check output stringification for an output in external repo.

You mean, for each remote separately? Please, clarify.

They do run, as the above TestCase is not changed at all. Or, did you mean this specific example?

Oh, I see, I was inattentive. And OutputLOCALBase not really used confused me.

You mean, for each remote separately? Please, clarify.

You don't need to do anything for each remote. You are only changing local one and thus only need to add tests for it. You can do that without touching existing tests at all and instead create two new test functions to check the specific changes you introduced.

@skshetry @Suor is saying that you've modified TestOutputLOCAL which is used in tests for other remotes, hence why the tests are failing and why not all tests are now running for those remotes. You need to not touch the old tests, but create a new one pytest-style test to test that stringification works properly for local output when you are in and outside of the repo. 🙂

Suor · 2019-11-16T10:28:04Z

dvc/output/local.py

+        root_dir = os.path.join(self.repo.root_dir, "")
+        curr_dir = os.getcwd()
+        if curr_dir.startswith(root_dir):


This pattern is used over and over in our code. This is probably a good time to extract this into some path_isin() utility.

Suor · 2019-11-16T10:31:29Z

tests/unit/output/test_local.py

+        o = self._get_output(wdir=work_dir)
+        self.assertEqual("path", str(o))
+
+    def test_str_if_workdir_outside_repo(self):


Now the three old tests are not tested for other remotes than local.

tests/unit/output/test_local.py

Suor · 2019-11-16T10:38:58Z

tests/unit/output/test_local.py

-        stage = Stage(self.dvc)
-        return self._get_cls()(stage, "path", cache=False)
+    def _get_output(self, wdir=None, path="path"):
+        stage = Stage(self.dvc, wdir=wdir if wdir else os.curdir)


The original issue this PR aims to solve was about outputs in external repo and thus outside it. Testing with stage of the same repo might not cover that or might break in the future.

Agreed. We could have two tests. One with Stage(dvc_repo) to test that it works correctly when in the repo, and one more test with Stage(erepo.dvc) to test that it works when outside of the repo. Should be enough.

Suor · 2019-11-16T10:40:13Z

tests/unit/output/test_local.py

+        o = self._get_output(wdir=work_dir)
+        self.assertEqual("path", str(o))
+
+    def test_str_if_workdir_outside_repo(self):


Considering all the test complications I suggest not touching existing ones, but adding a separate pytest-style test to check output stringification for an output in external repo.

tests/unit/utils/test_utils.py

pared · 2019-11-18T12:37:41Z

tests/unit/output/test_local.py

@@ -21,6 +23,35 @@ def test_save_missing(self):
                o.save()


+@pytest.mark.usefixtures("dvc_repo")


is this necessary?

If yes, then simply using as argument will do the job.

Looks like it's not necessary. Removing as erepo fixture already creates one.

pared · 2019-11-18T12:46:51Z

tests/unit/utils/test_utils.py

+        (os.path.join("path", "to")),
+        (os.path.join("path", "")),
+        (os.path.join("path")),
+    ],


I think all those test could be done with one, where parametrize child, path and expected result. But I guess it is a matter of taste.

Parametrizing for one line test is just a boiler plate.

I tried that one but I found parameterizing expected_result as well too confusing for myself (and, too much repetition). What do you think?

@skshetry I am ok with this as it is

I would still prefer several asserts in a single test. This is mere indirection, no upsides.

I also prefer having multiple asserts, but I think, this is easier to follow and understand.

pared

One question, one note, besides that LGTM.

Suor · 2019-11-16T17:50:34Z

tests/unit/output/test_local.py

+        o = self._get_output(wdir=work_dir)
+        self.assertEqual("path", str(o))
+
+    def test_str_if_workdir_outside_repo(self):


They do run, as the above TestCase is not changed at all. Or, did you mean this specific example?

Oh, I see, I was inattentive. And OutputLOCALBase not really used confused me.

You mean, for each remote separately? Please, clarify.

You don't need to do anything for each remote. You are only changing local one and thus only need to add tests for it. You can do that without touching existing tests at all and instead create two new test functions to check the specific changes you introduced.

Suor · 2019-11-18T13:25:39Z

dvc/utils/__init__.py

@@ -398,6 +398,14 @@ def relpath(path, start=os.curdir):
    return os.path.relpath(path, start)


+def is_child_path(child, parent):


I would call it path_isin() to be more in line with PathInfo.isin().

Renamed, thanks.

Suor · 2019-11-18T13:28:49Z

tests/unit/output/test_local.py

@@ -21,6 +23,35 @@ def test_save_missing(self):
                o.save()


+@pytest.mark.usefixtures("dvc_repo")


If yes, then simply using as argument will do the job.

Suor · 2019-11-18T13:33:02Z

tests/unit/utils/test_utils.py

+        (os.path.join("path", "to")),
+        (os.path.join("path", "")),
+        (os.path.join("path")),
+    ],


Parametrizing for one line test is just a boiler plate.

Suor · 2019-11-18T13:36:28Z

tests/unit/utils/test_utils.py

+def test_is_child_path_positive(parent):
+    child = os.path.join("path", "to", "folder")
+    assert is_child_path(child, parent)
+
+
+def test_is_child_on_same_path():
+    path = os.path.join("path", "to", "folder")
+    assert not is_child_path(child=path, parent=path)
+
+
+def test_is_child_on_different_path():
+    path1 = os.path.join("path", "to", "folder1")
+    path2 = os.path.join("path", "to", "folder2")
+
+    assert not is_child_path(path1, path2)


What about is_child_path('a/b/', 'a/b')? Comparing only things built with os.path.join() doesn't quite handle it.

@Suor I think that os.path.join(path, "") which is inside is_child_path handles this one.

Yes, I forgot to test about the case (which fails atm). I'll add that (using os.path.join(), as I am not sure about testing via raw strings.)

From what I understood, you are asking me to also test strings directly? But, that'd not be platform agnostic. That specific string test might need skipping on Windows (to add to that, the implementation is already quite dependent on os.path.join() although it claims it's purely lexical :) ).

Also, should there be a test for demonstrating support for PathInfo? (saw #2137, up-level references might fail on PathInfo, maybe normpath could help.)

@Suor, I refactored for PathInfo, as well as added few tests for absolute path and the case you just mentioned.

skshetry · 2019-11-18T16:46:28Z

I have 12 commits already. I'd prefer rebasing/squashing (or, is it not preferred for some reasons?)

shcheklein · 2019-11-18T16:51:08Z

I have 12 commits already. I'd prefer rebasing/squashing (or, is it not preferred for some reasons?)

@pared @MrOutis is the team policy flexible now? @skshetry I think it's totally up to you to manage commits in your branch in any way! And it's of course better for us in terms of merging if they look nice and clean :)

Suor

Looks like we reached cleanup. So there are many comments below, but they should go easy.

Suor · 2019-11-18T16:49:06Z

dvc/output/local.py

+from dvc.utils import relpath
+from dvc.utils import path_isin


We don't split imports like that anymore.

Suor · 2019-11-18T16:49:42Z

dvc/utils/__init__.py

+def path_isin(child, parent):
+    """Check if given `child` path is inside `parent`."""
+    parent = os.path.join(os.path.normpath(parent), "")
+    child = os.path.join(os.path.normpath(child), "")


The last os.path.join() is not needed.

Suor · 2019-11-18T16:50:40Z

dvc/utils/__init__.py

+    if parent != child and child.startswith(parent):
+        return True
+    return False


return child != parent and ...

Suor · 2019-11-18T16:54:41Z

tests/unit/utils/test_utils.py

+        (os.path.join("path", "to")),
+        (os.path.join("path", "")),
+        (os.path.join("path")),
+    ],


I would still prefer several asserts in a single test. This is mere indirection, no upsides.

Suor · 2019-11-18T16:57:32Z

tests/unit/utils/test_utils.py

+    assert not path_isin(child=path, parent=path)
+    assert not path_isin(child=path, parent=path2)
+    assert not path_isin(child=path2, parent=path)
+    assert not path_isin(child=path2, parent=path2)


Why are using pass by name here? I suggest treating those args as positional only.

I thought, it'd be less confusing here if I added the keywords. Removing ...

Suor · 2019-11-18T16:58:26Z

tests/unit/utils/test_utils.py

+    assert not path_isin(child=path2, parent=path2)
+
+
+def test_path_isin_on_different_path():


Test is about substring not some completely different path, test name is inpresize.

dvc/output/local.py

Suor · 2019-11-18T17:15:34Z

tests/unit/utils/test_utils.py

+    assert not path_isin(path1, path2)
+
+
+def test_path_isin_accepts_pathinfo():


I guess it won't work in Python 3.5 and earlier.

Oops, didn't test on Python3.5. Fixing ...

Suor · 2019-11-18T17:16:05Z

dvc/utils/__init__.py

@@ -398,6 +398,15 @@ def relpath(path, start=os.curdir):
    return os.path.relpath(path, start)


+def path_isin(child, parent):


Move this to dvc.utils.fs please.

I thought, dvc.utils.fs were mostly related to low-level like inodes and stuffs. Moving ...

@Suor, I have moved it to dvc.utils.fs and the tests as well.

ghost

No additional comments 👍

skshetry · 2019-11-18T23:56:36Z

@Suor, I have rebased, please review again.

Suor · 2019-11-19T11:31:45Z

dvc/utils/fs.py

+    """Check if given `child` path is inside `parent`."""
+
+    def normalize_path(path):
+        return os.path.normpath(fspath(path))


It should be fspath_py35() instead of fspath() here, since in Python 3.6+ you may pass Path-like object to os utils directly.

Changed, thanks.

tests/unit/utils/test_fs.py

Suor · 2019-11-19T11:33:53Z

tests/unit/utils/test_fs.py

+
+def test_path_isin_on_same_path():
+    path = os.path.join("path", "to", "folder")
+    path2 = os.path.join(path, "")


Suggested change

path2 = os.path.join(path, "")

path_with_sep = os.path.join(path, "")

efiop · 2019-11-19T16:48:57Z

@skshetry Please check tests, they are failing.

efiop · 2019-11-19T16:49:41Z

Ah, wait, those failures are probably unrelated. Looking into it...

efiop · 2019-11-19T23:59:50Z

Fixed the tests on master, everything should be smooth now. Sorry for the inconvenience 🙂

pared · 2019-11-20T08:53:21Z

When this patch is merged, we should create ticket for replacing startswith with path_isin throughout the code.

efiop · 2019-11-20T09:03:40Z

@Suor @MrOutis Guys, please take a look.

Suor

Looks good

efiop · 2019-11-20T09:34:08Z

Ah, I see @MrOutis already approved.

efiop · 2019-11-20T09:34:27Z

Thanks @skshetry ! 🎉

efiop · 2019-11-20T09:57:44Z

@skshetry Please don't forget to create a ticket for #2798 (comment) . 🙂

shcheklein requested review from efiop and Suor November 16, 2019 02:26

skshetry force-pushed the fix-str-rel-repo branch from b23a015 to a836705 Compare November 16, 2019 02:43

skshetry commented Nov 16, 2019

View reviewed changes

tests/unit/output/test_local.py Outdated Show resolved Hide resolved

skshetry commented Nov 16, 2019

View reviewed changes

tests/unit/output/test_local.py Outdated Show resolved Hide resolved

skshetry commented Nov 16, 2019

View reviewed changes

tests/unit/output/test_local.py Outdated Show resolved Hide resolved

skshetry marked this pull request as ready for review November 16, 2019 07:52

skshetry commented Nov 16, 2019

View reviewed changes

Suor suggested changes Nov 16, 2019

View reviewed changes

weekly-digest bot mentioned this pull request Nov 17, 2019

Weekly Digest (10 November, 2019 - 17 November, 2019) #2805

Closed

skshetry commented Nov 18, 2019

View reviewed changes

tests/unit/utils/test_utils.py Outdated Show resolved Hide resolved

skshetry commented Nov 18, 2019

View reviewed changes

tests/unit/utils/test_utils.py Outdated Show resolved Hide resolved

efiop requested review from pared, Suor and a user November 18, 2019 09:59

skshetry commented Nov 18, 2019

View reviewed changes

tests/unit/utils/test_utils.py Outdated Show resolved Hide resolved

pared reviewed Nov 18, 2019

View reviewed changes

Suor suggested changes Nov 18, 2019

View reviewed changes

ghost reviewed Nov 18, 2019

View reviewed changes

skshetry added 2 commits November 19, 2019 05:34

utils: implement path_isin to compare child and parent paths

68c7366

get/import: display path relative to temporary repo

f07a70a

skshetry force-pushed the fix-str-rel-repo branch from 704c420 to f07a70a Compare November 18, 2019 23:52

skshetry requested a review from Suor November 18, 2019 23:53

efiop requested a review from pared November 19, 2019 06:57

Suor suggested changes Nov 19, 2019

View reviewed changes

skshetry added 2 commits November 19, 2019 21:55

utils/fs: use fspath_py35 instead of fspath

58624e0

test: Refactor tests

83c8ab5

skshetry requested a review from Suor November 19, 2019 16:13

pared approved these changes Nov 20, 2019

View reviewed changes

efiop requested a review from a user November 20, 2019 09:03

Suor approved these changes Nov 20, 2019

View reviewed changes

efiop merged commit 91ac75d into iterative:master Nov 20, 2019

skshetry deleted the fix-str-rel-repo branch November 20, 2019 09:47

skshetry mentioned this pull request Nov 20, 2019

Use path_isin instead of startswith to compare filepaths #2823

Closed

weekly-digest bot mentioned this pull request Nov 24, 2019

Weekly Digest (17 November, 2019 - 24 November, 2019) #2841

Closed

		@@ -21,6 +23,35 @@ def test_save_missing(self):
		o.save()


		@pytest.mark.usefixtures("dvc_repo")

		@@ -398,6 +398,14 @@ def relpath(path, start=os.curdir):
		return os.path.relpath(path, start)


		def is_child_path(child, parent):

		from dvc.utils import relpath
		from dvc.utils import path_isin

		assert not path_isin(child=path2, parent=path2)


		def test_path_isin_on_different_path():

		assert not path_isin(path1, path2)


		def test_path_isin_accepts_pathinfo():

		@@ -398,6 +398,15 @@ def relpath(path, start=os.curdir):
		return os.path.relpath(path, start)


		def path_isin(child, parent):

	path2 = os.path.join(path, "")
	path_with_sep = os.path.join(path, "")

Display path relative to temporary repo when get/import-ing files #2798

Display path relative to temporary repo when get/import-ing files #2798

Conversation

skshetry commented Nov 16, 2019 • edited by jorgeorpinel Loading

Description

efiop commented Nov 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Suor Nov 16, 2019 • edited Loading

Choose a reason for hiding this comment

skshetry Nov 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop Nov 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Suor Nov 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry Nov 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pared left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry Nov 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry commented Nov 18, 2019

shcheklein commented Nov 18, 2019

Suor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skshetry Nov 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

skshetry commented Nov 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop commented Nov 19, 2019

efiop commented Nov 19, 2019

efiop commented Nov 19, 2019

pared commented Nov 20, 2019

efiop commented Nov 20, 2019

Suor left a comment

Choose a reason for hiding this comment

efiop commented Nov 20, 2019

efiop commented Nov 20, 2019

efiop commented Nov 20, 2019 • edited Loading

skshetry commented Nov 16, 2019 •

edited by jorgeorpinel

Loading

efiop commented Nov 16, 2019 •

edited

Loading

Suor Nov 16, 2019 •

edited

Loading

skshetry Nov 16, 2019 •

edited

Loading

efiop Nov 18, 2019 •

edited

Loading

Suor Nov 16, 2019 •

edited

Loading

skshetry Nov 18, 2019 •

edited

Loading

skshetry Nov 18, 2019 •

edited

Loading

skshetry Nov 18, 2019 •

edited

Loading

efiop commented Nov 20, 2019 •

edited

Loading