Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BF: Fix reported paths in ORA remote #5821

Merged
merged 2 commits into from Jul 26, 2021
Merged

Conversation

adswa
Copy link
Member

@adswa adswa commented Jul 26, 2021

In a local ORA remote, whereis only reported the object key:

(magma) adina@cpu12 in /data/group/psyinf/processing-workflow-tutorial on git:master
❱ git annex whereis fmriprep/sub-01/anat/sub-01_desc-brain_mask.nii.gz 
whereis fmriprep/sub-01/anat/sub-01_desc-brain_mask.nii.gz (3 copies) 
  	c034a599-5db4-4212-8c2f-5d1bfaa0af5c -- adina@juseless:/data/group/psyinf/processing-workflow-tutorial [here]
   	c564c916-629f-4bb8-8113-01a3b04d7a51 -- gin [gnode]
   	d8807e1f-25ef-4267-a5b9-698eae068b4c -- [output-storage]

  output-storage: Pv/77/MD5E-s422015--f1db473b0ef4bde6cd9bd1b6db9069ff.nii.gz/MD5E-s422015--f1db473b0ef4bde6cd9bd1b6db9069ff.nii.gz

(reported in #4979 (comment)).
This PR tries to fix this.

Fixes #4979

@adswa adswa requested a review from bpoldrack July 26, 2021 09:35
@adswa adswa added the semver-internal Changes only affect the internal API label Jul 26, 2021
@@ -1176,7 +1176,7 @@ def whereis(self, key):
self.annex.dirhash(key) + "/" + key + "/" + key

dsobj_dir, archive_path, key_path = self._get_obj_location(key)
return str(key_path) if self._local_io() \
return Path(dsobj_dir / str(key_path)) if self._local_io() \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a reason for the str() call on key_path. ;-)

This is ultimately written to stdout which then is read by git-annex. Meaning: Returning a Path object isn't useful here, as annexremote needs to convert it to string anyway. Plus: key_path should already be a Path if I'm not mistaken.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In short I think str(dsobj_dir / key_path) it should be.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this is all new to me. Would it then rather be

str(Path(dsobj_dir / key_path))

?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx! force pushed your suggestion.

@codecov
Copy link

codecov bot commented Jul 26, 2021

Codecov Report

Merging #5821 (6b2f177) into maint (3bb1e93) will decrease coverage by 60.59%.
The diff coverage is 0.00%.

❗ Current head 6b2f177 differs from pull request most recent head f289147. Consider uploading reports for the commit f289147 to get more accurate results
Impacted file tree graph

@@             Coverage Diff             @@
##            maint    #5821       +/-   ##
===========================================
- Coverage   90.14%   29.55%   -60.60%     
===========================================
  Files         300      297        -3     
  Lines       42353    42317       -36     
===========================================
- Hits        38180    12506    -25674     
- Misses       4173    29811    +25638     
Impacted Files Coverage Δ
datalad/distributed/ora_remote.py 20.57% <0.00%> (-11.80%) ⬇️
datalad/tests/test_api.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/support/digests.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/tests/test_base.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/tests/test_config.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/ui/tests/__init__.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/tests/test__main__.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/tests/test_strings.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/ui/tests/test_base.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/support/tests/utils.py 0.00% <0.00%> (-100.00%) ⬇️
... and 247 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3bb1e93...f289147. Read the comment docs.

@adswa adswa changed the title BF: In local ORA remotes, return the path to the dsobj dir too BF: Fix reported paths in ORA remote Jul 26, 2021
@adswa
Copy link
Member Author

adswa commented Jul 26, 2021

I have added a commit that hopefully fixes #4979

@adswa adswa force-pushed the bf-localorapath branch 2 times, most recently from 1e4b73f to 02043c9 Compare July 26, 2021 10:27
Copy link
Member

@bpoldrack bpoldrack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tried yet, but LGTM.

@@ -1172,11 +1172,11 @@ def whereis(self, key):
if isinstance(self.io, HTTPRemoteIO):
# display the URL for a request
# TODO: method of HTTPRemoteIO
return self.ria_store_url[4:] + "/annex/objects" + \
self.annex.dirhash(key) + "/" + key + "/" + key
return self.ria_store_url[4:] + "/annex/objects" + "/" + \
Copy link
Member

@bpoldrack bpoldrack Jul 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-critical nitpick: Why /annex/objects" + "/" instead of "/annex/objects/"? ;-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-critical nitpick: Why /annex/objects" + "/" instead of "/annex/objects/"? ;-)

Was mirroring your approach, in an attempt to be consistent :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

Yes, but I was adding to a variable. You are adding two constants. That's unnecessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, changed it

Copy link
Member

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear Lord, that I would live to see the day that this issues is resolved is divine! Thx much!

@mih mih linked an issue Jul 26, 2021 that may be closed by this pull request
@adswa adswa merged commit f94c089 into datalad:maint Jul 26, 2021
@adswa adswa deleted the bf-localorapath branch July 26, 2021 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-internal Changes only affect the internal API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Broken location reporting for ORA remote
3 participants