windows fix #22

frichtarik · 2024-04-17T09:37:05Z

this switches posix.DirEntry to os. DirEntry, which is os-independent proxy as suggested by https://docs.python.org/3/library/posix.html#module-posix
adding windows image to the GH workflow matrix
limiting codecov publisher step to one job from the matrix, no need to republish the same comment too many times

codecov-commenter · 2024-04-17T09:47:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (50b35fe) to head (f0a4386).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##            master       #22   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            7         7           
  Lines          350       348    -2     
  Branches        79        60   -19     
=========================================
- Hits           350       348    -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

frichtarik · 2024-04-22T13:44:21Z

@andhus #4 this is a draft on how to fix the windows compatibility (partially reverting/optimizing #11 )

i've set 5 tests to skip on Windows and adjusted some literals to be os agnostic
this causes tests to pass, circumventing some Windows peculiars, for the price of decreasing coverage (not sure why as that should be counted on Linux environment)

feel free to comment/adjust

frichtarik · 2024-04-22T13:57:00Z

@andhus could you please create a new release of the dirhash soon? currently there is a conflicting requirement of pathspec in scantree vs dirhash

with the renaming of workflow, job names changed and now PR expects 6 non-existing checks to finish

...which might fix the cross-environment issues? See: andhus/scantree#22

andhus · 2024-04-23T04:00:08Z

@andhus could you please create a new release of the dirhash soon? currently there is a conflicting requirement of pathspec in scantree vs dirhash

Oh, yes (!) I missed this one: https://github.com/andhus/scantree/blame/c50ed7dc85f30b2a9bcad2965e867087555cf29d/setup.py#L27

andhus · 2024-04-23T04:15:25Z

⬆️ https://github.com/andhus/dirhash-python/releases/tag/v0.4.0

andhus

Great stuff @frichtarik, thanks!

Please see a few small comments/questions.

andhus · 2024-04-23T04:32:45Z

tests/test_node.py

 import pytest

 from scantree import CyclicLinkedDir, DirNode, LinkedDir, RecursionPath
 from scantree.test_utils import get_mock_recursion_path


-def create_basic_entries(local_path):
+def create_basic_entries(local_path: str):


This is not a str, but a py.path.local gotten via the tmpdir fixture https://docs.pytest.org/en/6.2.x/tmpdir.html#the-tmpdir-fixture. The type hint here is misleading since local_path.join would do something very different if local_path is a string :)

(We should actually use the newer tmp_path fixture everytwhere, but in a separate PR!)

andhus · 2024-04-23T04:33:09Z

tests/test_node.py

    d1 = local_path.join("d1")
    d1.mkdir()
    f1 = local_path.join("f1")
    f1.write("file1")
-    local_path.join("ld1").mksymlinkto(d1)


Was this broken somehow (on windows)?

(Just curious)

yeah, i had to even try it on Win machine, that method is not present for some reason, so I've switched to os independent approach

andhus · 2024-04-23T04:36:00Z

tests/test_path.py

@@ -1,4 +1,4 @@
-from os import scandir
+from os import name, scandir, symlink


Detail (!) but feels a bit unnecessary to have a module-level variable named just name :) Maybe just import os and use os.name etc. for these, or from os import name as os_name, wdyt?

done, same for path >>os_path

andhus · 2024-04-23T04:36:59Z

tests/test_path.py

@@ -8,18 +8,19 @@
 from scantree.test_utils import assert_dir_entry_equal


-def create_basic_entries(local_path):
+def create_basic_entries(local_path: str):


Not a str see previous comment. (Would be great with type annotations, but in a separate PR :))

andhus · 2024-04-23T04:38:03Z

tests/test_scantree.py

@@ -163,6 +165,7 @@ def rp(relative):
        tree_empty_false_expected = DirNode(path=rp(""))
        assert tree_empty_false == tree_empty_false_expected

+    @pytest.mark.skipif(os.name == "nt", reason="Windows runs too slow for set limits")


Interesting, can take a look at this separately.

andhus · 2024-04-23T04:39:58Z

src/scantree/test_utils.py

@@ -4,20 +4,37 @@
 from ._path import DirEntryReplacement, RecursionPath


+def convert_path(path: str) -> str:


Cool, I guess we could have replaced the plain string paths with os.path.join(...) as well, but this works well 👍

yeah, but it was not sure it will work for already complete paths stored as strings

andhus · 2024-04-23T04:43:18Z

src/scantree/test_utils.py

        ("inode", {}),
-    ]:
+    ]
+    if os.name != "nt":


Yeah I guess it is expected and that stat should be ignore in Windows https://stackoverflow.com/a/34572372

While at it, could you please update _path.DirEntryReplacement.__eq__ to ignore stat as well in the comparison?

andhus · 2024-04-23T04:47:15Z

with the renaming of workflow, job names changed and now PR expects 6 non-existing checks to finish

Yes, will update those after the merge of this PR 👍

frichtarik · 2024-05-01T09:08:20Z

sorry for the delay, I got to switch employer i am was working for and switch dev machine 🤦
after last fixes, there is one test that fails on LIN but not WIN, going to fix that later today and we should be good to go

bossanova808 · 2024-05-22T02:15:30Z

Would the lack of this PR be why using this in two WSL environments seems not to work at all?

Dirhashes seem to be always different across the environments (each is pointing at a Windows drive, e.g.: /mnt/e/Git/Kodi/repository.bossanova808

frichtarik · 2024-06-22T11:05:12Z

.pre-commit-config.yaml

@@ -1,12 +1,7 @@
 exclude: "src/scantree/_version.py"
 repos:
-  - repo: https://github.com/pre-commit/mirrors-prettier


removed mirrors-prettier as the repo got deprecated (underlying prettier changes breaking the project altogether)

frichtarik · 2024-06-22T11:07:53Z

It took longer than expected, but i'm finally done

racinmat · 2024-07-09T11:51:28Z

@andhus how does it look with the merging?

frichtarik force-pushed the windows_fix branch from 433f657 to a375255 Compare April 17, 2024 12:13

frichtarik marked this pull request as draft April 17, 2024 12:51

frichtarik force-pushed the windows_fix branch 13 times, most recently from 6a4c859 to 851febc Compare April 22, 2024 13:30

windows fix

8b641db

frichtarik force-pushed the windows_fix branch from 851febc to 8b641db Compare April 22, 2024 13:36

frichtarik marked this pull request as ready for review April 22, 2024 13:41

frichtarik mentioned this pull request Apr 22, 2024

not working on windows andhus/dirhash-python#18

Open

bossanova808 added a commit to bossanova808/repository.bossanova808 that referenced this pull request Apr 22, 2024

Revert to dirhash, there is a Windows fix pending for it...

f8b1f3e

...which might fix the cross-environment issues? See: andhus/scantree#22

andhus reviewed Apr 23, 2024

View reviewed changes

racinmat mentioned this pull request Apr 25, 2024

[FeatureRequest]: we can increase the scantree version soon avast/wanna-ml#130

Open

review fixes

fd0612a

frichtarik force-pushed the windows_fix branch from d045044 to fd0612a Compare May 1, 2024 08:59

frichtarik added 3 commits June 22, 2024 10:17

Merge branch 'master' into windows_fix

dc94f62

inode

76e4b77

pre-commit updates

a071a04

frichtarik force-pushed the windows_fix branch from 0771bf1 to 7226cd9 Compare June 22, 2024 10:47

coverage

f0a4386

frichtarik force-pushed the windows_fix branch from 9da0673 to f0a4386 Compare June 22, 2024 11:00

frichtarik commented Jun 22, 2024

View reviewed changes

frichtarik requested a review from andhus June 22, 2024 11:08

mparanjpe mentioned this pull request Jul 4, 2024

Allow scantree to work on Windows #24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

windows fix #22

windows fix #22

frichtarik commented Apr 17, 2024 •

edited

Loading

codecov-commenter commented Apr 17, 2024 •

edited by codecov bot

Loading

frichtarik commented Apr 22, 2024

frichtarik commented Apr 22, 2024 •

edited

Loading

andhus commented Apr 23, 2024

andhus commented Apr 23, 2024

andhus left a comment

andhus Apr 23, 2024

frichtarik May 1, 2024

andhus Apr 23, 2024

andhus Apr 23, 2024

frichtarik May 1, 2024

andhus Apr 23, 2024

frichtarik May 1, 2024

andhus Apr 23, 2024

frichtarik May 1, 2024

andhus Apr 23, 2024

andhus Apr 23, 2024

frichtarik May 1, 2024

andhus Apr 23, 2024

andhus Apr 23, 2024

andhus commented Apr 23, 2024

frichtarik commented May 1, 2024

bossanova808 commented May 22, 2024

frichtarik Jun 22, 2024

frichtarik commented Jun 22, 2024

racinmat commented Jul 9, 2024 •

edited

Loading

		@@ -1,4 +1,4 @@
		from os import scandir
		from os import name, scandir, symlink

		@@ -4,20 +4,37 @@
		from ._path import DirEntryReplacement, RecursionPath


		def convert_path(path: str) -> str:

windows fix #22

Are you sure you want to change the base?

windows fix #22

Conversation

frichtarik commented Apr 17, 2024 • edited Loading

codecov-commenter commented Apr 17, 2024 • edited by codecov bot Loading

Codecov Report

frichtarik commented Apr 22, 2024

frichtarik commented Apr 22, 2024 • edited Loading

andhus commented Apr 23, 2024

andhus commented Apr 23, 2024

andhus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andhus commented Apr 23, 2024

frichtarik commented May 1, 2024

bossanova808 commented May 22, 2024

Choose a reason for hiding this comment

frichtarik commented Jun 22, 2024

racinmat commented Jul 9, 2024 • edited Loading

frichtarik commented Apr 17, 2024 •

edited

Loading

codecov-commenter commented Apr 17, 2024 •

edited by codecov bot

Loading

frichtarik commented Apr 22, 2024 •

edited

Loading

racinmat commented Jul 9, 2024 •

edited

Loading