Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added yandex.disk support and test file #6667

Merged
merged 4 commits into from
Jun 28, 2023

Conversation

blacky-i
Copy link
Contributor

Fixes #6666
/black

Description

Updated download_url to support internally yandex.disk hosting

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: BlackyI <cheivan@mail.ru>
wyli added 2 commits June 28, 2023 15:45
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
…-support

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: BlackyI <cheivan@mail.ru>
@wyli
Copy link
Member

wyli commented Jun 28, 2023

/build

@wyli wyli enabled auto-merge (squash) June 28, 2023 15:13
@wyli wyli merged commit d4b9552 into Project-MONAI:dev Jun 28, 2023
30 of 35 checks passed

YANDEX_MODEL_URL = (
"https://cloud-api.yandex.net/v1/disk/public/resources/download?"
"public_key=https%3A%2F%2Fdisk.yandex.ru%2Fd%2Fxs0gzlj2_irgWA"
Copy link
Member

@wyli wyli Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test will run regularly by testing pipelines, do you think this URL and the file will be available all the time for those pipelines? @blacky-i (currently removed 7cb71a0)

Copy link
Contributor Author

@blacky-i blacky-i Jun 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two risks in this url:

  1. API deprecation (which is very unlikely)
  2. This little file is on my personal yandex disk, I will not delete it.

Copy link
Member

@wyli wyli Jun 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, it seems when there are concurrent downloadings from multiple jobs, it becomes unavailable, for example:

https://github.com/Project-MONAI/MONAI/actions/runs/5403919298/jobs/9817414158

======================================================================
ERROR: test_verify (tests.test_download_url_yandex.TestDownloadUrlYandex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/MONAI/MONAI/tests/test_download_url_yandex.py", line 34, in test_verify
    download_url(url=YANDEX_MODEL_URL, filepath=os.path.join(tempdir, "model.pt"))
  File "/home/runner/work/MONAI/MONAI/monai/apps/utils.py", line 208, in download_url
    with urlopen(url) as response:
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/opt/hostedtoolcache/Python/3.8.17/x64/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: NOT FOUND

----------------------------------------------------------------------

anyway I think it's now skipped and is not a critical issue because it'll not impact the other downloading capability..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[PullRequest] Adding Yandex.Disk download support
2 participants