Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix path and URL handling in ORA remote and associated tests #5136

Merged
merged 3 commits into from Nov 9, 2020

Conversation

mih
Copy link
Member

@mih mih commented Nov 8, 2020

This fixes datalad/datalad-ria#18 but it does not make the relevant test complete due to #5134. However, with these changes, I can manually create RIA siblings, push them, clone them, and get data from them -- on windows.

(base) C:\Users\mih\TMP>datalad create someds
[INFO] Creating a new annex repo at C:\Users\mih\TMP\someds
[INFO] Detected a filesystem without fifo support.
[INFO] Disabling ssh connection caching.
[INFO] Detected a crippled filesystem.
[INFO] Scanning for unlocked files (this may take some time)
[INFO] Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
[INFO] Switched to branch 'adjusted/master(unlocked)'
create(ok): C:\Users\mih\TMP\someds (dataset)

(base) C:\Users\mih\TMP>datalad -C someds create-sibling-ria -s store ria+file://C/Users/mih/TMP/RIA
[INFO] create siblings 'store' and 'store-storage' ...
[INFO] Fetching updates for Dataset("C:\Users\mih\TMP\someds")
[INFO] Configure additional publication dependency on "store-storage"
create-sibling-ria(ok): C:\Users\mih\TMP\someds (dataset)

(base) C:\Users\mih\TMP>echo 123 > someds\dummy.txt

(base) C:\Users\mih\TMP>datalad -C someds save
add(ok): dummy.txt (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)

(base) C:\Users\mih\TMP>datalad -C someds push --to store
copy(ok): dummy.txt (file) [to store-storage...]
publish(ok): . (dataset) [refs/heads/master->store:refs/heads/master [new branch]]
publish(ok): . (dataset) [refs/heads/git-annex->store:refs/heads/git-annex [new branch]]

(base) C:\Users\mih\TMP>cat someds\.datalad\config
[datalad "dataset"]
        id = 7b95bb8c-750f-4e2e-8e91-a083af8b2b13

(base) C:\Users\mih\TMP>datalad clone ria+file:///C/Users/mih/TMP/RIA#7b95bb8c-750f-4e2e-8e91-a083af8b2b13 from_ria
[INFO] Detected a filesystem without fifo support.
[INFO] Disabling ssh connection caching.
[INFO] Detected a crippled filesystem.
[INFO] Scanning for unlocked files (this may take some time)
[INFO] Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
[INFO] Switched to branch 'adjusted/master(unlocked)'
[INFO] Remote origin cannot currently be accessed.
[INFO] Configure additional publication dependency on "store-storage"
configure-sibling(ok): . (sibling)
install(ok): C:\Users\mih\TMP\from_ria (dataset)
action summary:
  configure-sibling (ok: 1)
  install (ok: 1)

at this point it requires a manual intervention that should not be necessary (#5134).

(base) C:\Users\mih\TMP>git -C from_ria annex enableremote store-storage
enableremote store-storage ok
(recording state in git...)

but afterwards we are in business

(base) C:\Users\mih\TMP>datalad -C from_ria get dummy.txt
get(ok): dummy.txt (file) [from store-storage...]

(base) C:\Users\mih\TMP>cat from_ria\dummy.txt
123

@codecov
Copy link

codecov bot commented Nov 8, 2020

Codecov Report

Merging #5136 (acf6779) into maint (ace8110) will decrease coverage by 29.24%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##            maint    datalad/datalad#5136       +/-   ##
===========================================
- Coverage   89.95%   60.71%   -29.25%     
===========================================
  Files         293      126      -167     
  Lines       40909    18907    -22002     
===========================================
- Hits        36800    11479    -25321     
- Misses       4109     7428     +3319     
Impacted Files Coverage Δ
datalad/distributed/ora_remote.py 20.88% <0.00%> (-10.37%) ⬇️
datalad/plugin/addurls.py 17.32% <0.00%> (-82.39%) ⬇️
datalad/interface/rerun.py 14.52% <0.00%> (-82.10%) ⬇️
datalad/local/copy_file.py 15.08% <0.00%> (-78.88%) ⬇️
datalad/plugin/export_archive.py 24.65% <0.00%> (-75.35%) ⬇️
datalad/interface/add_archive_content.py 19.33% <0.00%> (-70.76%) ⬇️
datalad/metadata/search.py 19.50% <0.00%> (-69.19%) ⬇️
...ad/metadata/extractors/frictionless_datapackage.py 27.08% <0.00%> (-68.75%) ⬇️
datalad/metadata/extractors/datalad_rfc822.py 30.90% <0.00%> (-67.28%) ⬇️
datalad/plugin/add_readme.py 29.31% <0.00%> (-63.80%) ⬇️
... and 252 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ace8110...acf6779. Read the comment docs.

A store's base path was wrongly assumend to be always in POSIX convention,
which prevents operation in non-POSIX platforms.

This change treats the base path as being a PurePosixPath (when derived
from the base URL), but explicitly converts it into a platform Path
prior to any local file system operations.
The latter is not capable of producing file URLs on Windows that work
with Git and/or git-annex.
@yarikoptic yarikoptic merged commit 4434043 into datalad:maint Nov 9, 2020
@mih
Copy link
Member Author

mih commented Nov 9, 2020

Thx @bpoldrack and @yarikoptic !

@mih mih deleted the bf-5085 branch November 9, 2020 13:25
mih pushed a commit that referenced this pull request Dec 4, 2020
Fix path and URL handling in ORA remote and associated tests
@yarikoptic yarikoptic added this to the 0.13.6 milestone Dec 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform-windows Issue concerned with Windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid path handling test_ria_postclonecfg
3 participants