Cannot access https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth for pretrained model #5614

Chenyangzh · 2023-06-20T10:27:52Z

Describe the issue:

Hi, I tried to run official demo darts nas, but met an issue.

Traceback (most recent call last):
File "darts.py", line 74, in
darts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)
File "/root/miniconda3/lib/python3.7/site-packages/nni/nas/hub/pytorch/nasnet.py", line 888, in load_searched_model
weight_file = load_pretrained_weight(name, download=download, progress=progress)
File "/root/miniconda3/lib/python3.7/site-packages/nni/nas/hub/pytorch/utils/pretrained.py", line 58, in load_pretrained_weight
load_or_download_file(local_path, url, **kwargs)
File "/root/miniconda3/lib/python3.7/site-packages/nni/common/blob_utils.py", line 85, in load_or_download_file
raise RuntimeError('Invalid hash value (expected "{}", got "{}")'.format(hash_prefix, digest))
RuntimeError: Invalid hash value (expected "5465b0d2", got "ce49989c8f6ef11f2b2192c41c62a8701f3057fd0c25a27cf6b66253b92496ec")

I checked the source code and found that the reason is cannot access pertrained searched model. Any solution?

Environment:

NNI version: 2.10.1
Training service (local|remote|pai|aml|etc): local
Client OS:
Server OS (for remote mode only):
Python version: 3.7
PyTorch/TensorFlow version:
Is conda/virtualenv/venv used?: conda
Is running in Docker?: no

Configuration:

Experiment config (remember to remove secrets!):
Search space:

Log message:

nnimanager.log:
dispatcher.log:
nnictl stdout and stderr:

How to reproduce it?:

matluster · 2023-06-21T03:22:37Z

It appears that the blob access is set to private for some reason.

Please try again as I just turned on public access.

jungin500 · 2023-06-25T12:15:36Z

It seems that blob is still not accessible for me, too.
Would you check once again? thanks in advance.

URL: https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth
cURL Output:

HTTP/1.1 409 Public access is not permitted on this storage account.
Content-Length: 248
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 5bf6c42b-801e-009b-7e5e-a7230c000000
x-ms-version: 2009-09-19
Date: Sun, 25 Jun 2023 12:14:51 GMT

<?xml version="1.0" encoding="utf-8"?><Error><Code>PublicAccessNotPermitted</Code><Message>Public access is not permitted on this storage account.
RequestId:5bf6c42b-801e-009b-7e5e-a7230c000000
Time:2023-06-25T12:14:51.6472750Z</Message></Error>

ultmaster · 2023-06-28T02:47:49Z

Turned on again. Please try again.

jungin500 · 2023-06-28T02:56:04Z

Turned on again. Please try again.

Thank you. I have confirmed that the link is working now.

Lijiaoa · 2023-06-30T02:13:23Z

Could you close it? @Chenyangzh

jungin500 · 2023-06-30T11:52:59Z

TL;DR - This issue has not been resolved yet. I'm only going to provide a mirror as a workaround until the issue is resolved.

Although I've been periodically checking, it's evident that this issue has not been resolved permantly.

To help future viewers who may encounter this issue, I'm currently hosting an ephemeral mirror server for those weights. (Please note that the server may not remain available permanently!)

If the MS blob server goes down again, you can use my mirror by utilizing the following code:

#
# Before running any NNI script,
# add the following script at the TOP of the Python file.
#

import nni.common.blob_utils
nni.common.blob_utils.NNI_BLOB = 'https://repo.dailylime.kr/mirror/nni'

It will monkey-patch the blob url and download weights from my server.

matluster · 2023-07-01T01:59:43Z

Thanks @jungin500.

I think Microsoft has some security regulations that is preventing us from making the blob public. They might have periodically-run script that closes the public access if the public access is not officially authorized. To make it officially authorized, we might need to go through a tedious review process which none of us are willing to push forward.

Given the current bandwidth and financial support on this NNI project, any solution that requires much laboring (either code-work or paperwork), or requires constantly paying a certain amount of money, IS NOT acceptable to us. Our best shot is to find some 3rd-party blob services (that are better to be free) to host these files. It's unfortunate that we have to repeatedly turn on the public access before a workaround is implemented.

xinlnix · 2023-08-11T02:37:44Z

I have the same issue agin. Could you open the access again? @ultmaster

xinlnix · 2023-08-11T02:39:40Z

Could you please open the aceess? Thanks @matluster

ShabiAta · 2023-09-22T07:12:20Z

Cannot download pertained models for nni (permission denied)

flower997 · 2023-09-22T07:30:06Z

I also want to download datrs-v2.I have this question.There is an error
"/home/xia-group-0/.cache/nni/nashub/darts-v2-5465b0d2.pth" does not exist. Downloading "https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth"
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'nni.blob.core.windows.net'. (_ssl.c:1129)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nni.blob.core.windows.net', port=443): Max retries exceeded with url: /nashub/darts-v2-5465b0d2.pth (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'nni.blob.core.windows.net'. (_ssl.c:1129)")))

jungin500 · 2023-09-29T11:33:48Z

I also want to download datrs-v2.I have this question.There is an error "/home/xia-group-0/.cache/nni/nashub/darts-v2-5465b0d2.pth" does not exist. Downloading "https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth" ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'nni.blob.core.windows.net'. (_ssl.c:1129) requests.exceptions.SSLError: HTTPSConnectionPool(host='nni.blob.core.windows.net', port=443): Max retries exceeded with url: /nashub/darts-v2-5465b0d2.pth (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'nni.blob.core.windows.net'. (_ssl.c:1129)")))

@flower997 As the error Hostname mismatch says, this relates to the network issue of your infrastructure. can you check if you can access another HTTPS-enabled websites using curl or wget?

I think It's highly possible that firewall or router is showing you an captive portal page so you can't access other pages, too. (Although the download server is down, this certificate issue shouldn't be happend at all.)

In a meantime, the download server is currently down, so I recommend you using my public clone of the weight server.

AmT42 · 2023-12-02T12:35:17Z

Got the same error here:
"""[2023-12-02 13:29:43] "/home/ubuntulinux/.cache/nni/nashub/darts-v2-5465b0d2.pth" does not exist. Downloading "https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth"
[2023-12-02 13:29:43] azcopy is not installed. Fall back to use requests.
100%|██████████| 248/248 [00:00<00:00, 930kB/s]
RuntimeError: Invalid hash value (expected "5465b0d2", got "02cb5a7bcd87cf5a349e6adece9b923198cd3a5e644baf73ebe193e6a506c89c") for /home/ubuntulinux/.cache/nni/nashub/darts-v2-5465b0d2.pth. Please delete the file and try re-downloading.
"""

While I can wget other urls

AhmadrezaHadi · 2023-12-08T22:26:47Z

Same for me. Is there any fix for it?

AmT42 · 2023-12-08T23:10:52Z

Add this at the very beggening of your notebook or py file

import nni.common.blob_utils
nni.common.blob_utils.NNI_BLOB = 'https://repo.dailylime.kr/mirror/nni'

DiegoC1212 · 2024-03-04T10:52:53Z

Hi, the mirror server appear to be unreachable. Is there a way to download the pre trained models?

jungin500 · 2024-03-04T11:39:18Z

Hi, the mirror server appear to be unreachable. Is there a way to download the pre trained models?

Hello, I've fixed my mirror server. can you access the server? It should be available now.

DiegoC1212 · 2024-03-04T12:09:23Z

Yea, working now thank you!

ranranrannervous · 2024-04-17T12:11:33Z

看来我仍然无法访问 blob。你会再检查一次吗？提前致谢。

网址：https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth
cURL 输出：

HTTP/1.1 409 Public access is not permitted on this storage account.
Content-Length: 248
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 5bf6c42b-801e-009b-7e5e-a7230c000000
x-ms-version: 2009-09-19
Date: Sun, 25 Jun 2023 12:14:51 GMT

<?xml version="1.0" encoding="utf-8"?><Error><Code>PublicAccessNotPermitted</Code><Message>Public access is not permitted on this storage account.
RequestId:5bf6c42b-801e-009b-7e5e-a7230c000000
Time:2023-06-25T12:14:51.6472750Z</Message></Error>

我也一样

ultmaster · 2024-05-23T07:08:23Z

I've uploaded the nasbenchmark and nashub data to my personal Google drive because I was told that the blob was about to be shutdown.

For anyone in need, please download it from my Google drive here

Bonytu assigned ultmaster Jun 21, 2023

xinlnix mentioned this issue Aug 11, 2023

Can not download Tar when Build from Source #5651

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot access https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth for pretrained model #5614

Cannot access https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth for pretrained model #5614

Chenyangzh commented Jun 20, 2023

matluster commented Jun 21, 2023

jungin500 commented Jun 25, 2023

ultmaster commented Jun 28, 2023

jungin500 commented Jun 28, 2023

Lijiaoa commented Jun 30, 2023

jungin500 commented Jun 30, 2023

matluster commented Jul 1, 2023 •

edited

xinlnix commented Aug 11, 2023

xinlnix commented Aug 11, 2023

ShabiAta commented Sep 22, 2023

flower997 commented Sep 22, 2023

jungin500 commented Sep 29, 2023

AmT42 commented Dec 2, 2023

AhmadrezaHadi commented Dec 8, 2023

AmT42 commented Dec 8, 2023

DiegoC1212 commented Mar 4, 2024

jungin500 commented Mar 4, 2024

DiegoC1212 commented Mar 4, 2024

ranranrannervous commented Apr 17, 2024

ultmaster commented May 23, 2024

Cannot access https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth for pretrained model #5614

Cannot access https://nni.blob.core.windows.net/nashub/darts-v2-5465b0d2.pth for pretrained model #5614

Comments

Chenyangzh commented Jun 20, 2023

matluster commented Jun 21, 2023

jungin500 commented Jun 25, 2023

ultmaster commented Jun 28, 2023

jungin500 commented Jun 28, 2023

Lijiaoa commented Jun 30, 2023

jungin500 commented Jun 30, 2023

matluster commented Jul 1, 2023 • edited

xinlnix commented Aug 11, 2023

xinlnix commented Aug 11, 2023

ShabiAta commented Sep 22, 2023

flower997 commented Sep 22, 2023

jungin500 commented Sep 29, 2023

AmT42 commented Dec 2, 2023

AhmadrezaHadi commented Dec 8, 2023

AmT42 commented Dec 8, 2023

DiegoC1212 commented Mar 4, 2024

jungin500 commented Mar 4, 2024

DiegoC1212 commented Mar 4, 2024

ranranrannervous commented Apr 17, 2024

ultmaster commented May 23, 2024

matluster commented Jul 1, 2023 •

edited