Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_local_file_url() broken on windows (i.e. for use as annex input) #3974

Closed
mih opened this issue Dec 30, 2019 · 1 comment
Closed

get_local_file_url() broken on windows (i.e. for use as annex input) #3974

mih opened this issue Dec 30, 2019 · 1 comment
Assignees
Labels
fix-implemented A fix is available, but has not been merged or released, yet. platform-windows Issue concerned with Windows

Comments

@mih
Copy link
Member

mih commented Dec 30, 2019

In short:

======================================================================
ERROR: datalad.tests.test_utils_testrepos.test_BasicAnnexTestRepo_random_location_generated
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\mih\Miniconda3\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\Users\mih\code\datalad\datalad\tests\test_utils_testrepos.py", line 42, in test_BasicAnnexTestRepo_random_location_generated
    _test_BasicAnnexTestRepo(None)  # without explicit path -- must be generated
  File "C:\Users\mih\code\datalad\datalad\tests\test_utils_testrepos.py", line 28, in _test_BasicAnnexTestRepo
    trepo.create()
  File "C:\Users\mih\code\datalad\datalad\tests\utils_testrepos.py", line 90, in create
    self.populate()
  File "C:\Users\mih\code\datalad\datalad\tests\utils_testrepos.py", line 117, in populate
    self.repo.add_url_to_file("test-annex.dat", fileurl)
  File "C:\Users\mih\code\datalad\datalad\support\gitrepo.py", line 231, in newfunc
    return func(self, file_new, *args, **kwargs)
  File "C:\Users\mih\code\datalad\datalad\support\annexrepo.py", line 1895, in add_url_to_file
    % (url, str(out_json)))
datalad.support.exceptions.CommandError: CommandError: command 'addurl'
Error, annex reported failure for addurl (url='file:///C%3A/Users/mih/AppData/Local/Temp/datalad_temp_testrepo_fpl3aaoz'): {'command': 'addurl', 'success': False, 'error-messages': ['  download failed: /C:/Users/mih/AppData/Local/Temp/datalad_temp_testrepo_fpl3aaoz: openBinaryFile: invalid argument (Invalid argument)'], 'file': 'test-annex.dat'}

This function is used at the core of the testrepo setup, hence its breakage is a key reason for the disfunctionality of a large part of our test battery on windows.

It took me forever to find this, because the tests that cover it were disabled 4 years ago c53298e

It is not clear to me how a working file:// URL could look like. According to https://en.wikipedia.org/wiki/File_URI_scheme#Windows it is not too far off. Maybe just one slash less would do it. But I cannot seem to make anything work with git-annex directly:

> git -c annex.security.allowed-url-schemes=file annex addurl file://./c:/Windows/clock.avi
addurl file://./c:/Windows/clock.avi

  download failed: /c:/Windows/clock.avi: openBinaryFile: invalid argument (Invalid argument)
failed
git-annex: addurl: 1 failed

>git -c annex.security.allowed-url-schemes=file annex addurl file://c$/Windows/clock.avi
addurl file://c$/Windows/clock.avi

  download failed: /Windows/clock.avi: openBinaryFile: does not exist (No such file or directory)
failed
git-annex: addurl: 1 failed

>git -c annex.security.allowed-url-schemes=file annex addurl file:///c$/Windows/clock.avi
addurl file:///c$/Windows/clock.avi

  download failed: /c$/Windows/clock.avi: openBinaryFile: does not exist (No such file or directory)
failed
git-annex: addurl: 1 failed

>git -c annex.security.allowed-url-schemes=file annex addurl file://localhost/c$/Windows/clock.avi
addurl file://localhost/c$/Windows/clock.avi

  download failed: /c$/Windows/clock.avi: openBinaryFile: does not exist (No such file or directory)
failed
git-annex: addurl: 1 failed

>git -c annex.security.allowed-url-schemes=file annex addurl file://localhost/c:/Windows/clock.avi
addurl file://localhost/c:/Windows/clock.avi

  download failed: /c:/Windows/clock.avi: openBinaryFile: invalid argument (Invalid argument)
failed
git-annex: addurl: 1 failed
@mih mih added the platform-windows Issue concerned with Windows label Dec 30, 2019
@mih
Copy link
Member Author

mih commented Dec 30, 2019

OK, figured it out:

(base) C:\Users\mih\code\datalad\this>git -c annex.security.allowed-url-schemes=file annex addurl file:///Windows/Notepad.exe
addurl file:///Windows/Notepad.exe
(to _Windows_Notepad.exe) ok
(recording state in git...)

(base) C:\Users\mih\code\datalad\this>git -c annex.security.allowed-url-schemes=file annex addurl file://C/Windows/Notepad.exe
addurl file://C/Windows/Notepad.exe
(to C_Windows_Notepad.exe) ok
(recording state in git...)

i.e. tripple-slash when no drive letter, double-slash otherwise, and no colons

@mih mih self-assigned this Dec 30, 2019
mih added a commit to mih/datalad that referenced this issue Dec 31, 2019
mih added a commit to mih/datalad that referenced this issue Dec 31, 2019
mih added a commit to mih/datalad that referenced this issue Dec 31, 2019
This changes removes the ability to deal with filenames that have
trailing slashes on non-windows systems for reasons of implementation
simplicity. If there are such file names we might reconsider, but I
doubt that any other part of datalad can handle those.

Fixes dataladgh-3974
mih added a commit to mih/datalad that referenced this issue Dec 31, 2019
mih added a commit to mih/datalad that referenced this issue Dec 31, 2019
@mih mih added the fix-implemented A fix is available, but has not been merged or released, yet. label Dec 31, 2019
@mih mih closed this as completed in f677e06 Jan 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix-implemented A fix is available, but has not been merged or released, yet. platform-windows Issue concerned with Windows
Projects
None yet
Development

No branches or pull requests

1 participant