HTTP Error 403: rate limit exceeded when loading model #4156

ichamatidis · 2021-07-06T15:41:36Z

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

import torch
model = torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

Expected behavior

Load the model

Environment

Google Colab and also local with the latest pytorch and torchvision installed

Error Output

HTTPError Traceback (most recent call last)

in ()
1 import torchvision
2
----> 3 model = torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

8 frames

/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648 def http_error_default(self, req, fp, code, msg, hdrs):
--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)
650
651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: rate limit exceeded

oke-aditya · 2021-07-06T18:01:37Z

I'm able to download on my local machine. With the following command

import torch
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True)

Also with

import torch
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet50', pretrained=True)

Can you give one more try @ichamatidis ?

ichamatidis · 2021-07-06T18:17:39Z

I tried it with both versions that you mentioned, still the same result in Google Colab and locally. I provide bellow the error output form the 0.10.0 version from Google Colab.

Error Output

HTTPError Traceback (most recent call last)

in ()
1 import torch
2
----> 3 model = torch.hub.load('pytorch/vision:v0.10.0','resnet50', pretrained=True)

8 frames

/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648 def http_error_default(self, req, fp, code, msg, hdrs):
--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)
650
651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: rate limit exceeded

oke-aditya · 2021-07-06T18:37:44Z

True. I can reproduce this over Google Colab. Not locally though

Here is link

Unsure why colab should do that.

But you can access pretrained model from
rnet = torchvision.models.resnet50(pretrained=True)

ichamatidis · 2021-07-06T19:11:35Z

Yeah it's a weird bug and must be recent(less than 2 weeks cause it was working normally before). Does your local machine has Windows or Linux?

oke-aditya · 2021-07-06T19:46:33Z

Linux Ubuntu 20.04.2

ichamatidis · 2021-07-07T00:21:03Z

I have Ubuntu 21.04, so I assume it's some connection error. Thank you for your advise :)

ptrblck · 2021-07-07T06:15:44Z

Issue is also observed here and I'm also able to reproduce it using Ubuntu20.04.

pmeier · 2021-07-07T06:51:56Z

This seems to be a bug in torch.hub with pytorch/pytorch#56138 as possible offender. I can't reproduce the issue with torch==1.8.0 which does not contain this commit. cc @cyyever @ailzhang

ailzhang · 2021-07-07T08:16:29Z

Yea I think it's introduced in that PR cc: @cyyever. It's probably hitting the rate limit set by github. My guess is it's only reproducible on CI machines/Google Colab since those machines might send a lot of API requests from the same public IP address so github only errors out on those. Our personal dev machines didn't hit the limit so it's not reproducible there.
If we still don't want to introduce dependency on system git, we probably need to either find another way to validate branch/tag or somehow workaround the rate limit.

AK391 · 2021-07-07T17:13:17Z

torch==1.8.0

can confirm using torch==1.8.0 and restarting runtime in colab works

Vernerrf · 2021-07-08T00:05:20Z

Hi everyone,
With me the error happens with any model used. And even resetting the collab the problem persists.

pmeier · 2021-07-08T05:13:35Z

@ailzhang I can reproduce this locally and @NicolasHug seems to have hit the same in #4157. So it is not only CI machines and Colab.

Bumping priority due to user activity.

SomTambe · 2021-07-15T13:13:12Z

I'm getting the same error even while trying to run the torch hub models on docker containers. In my opinion, that should not happen.
The system is running Ubuntu 20.04.

cyyever · 2021-07-16T06:07:03Z

This is because we are using GitHub API to check the existence of the repo. I have submitted a PR to ignore HTTP error 403. To work around this error temporarily, you can delete line 163 on torch/hub.py

_validate_not_a_forked_repo(repo_owner, repo_name, branch)

pmeier · 2021-07-16T06:11:06Z

pytorch/pytorch#61761 for reference.

NicolasHug · 2021-07-16T08:19:58Z

Also @ichamatidis :

torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

This should be either vision:v0.9.0 or vision:v0.10.0, as torchvision's version 1.9 doesn't exist.

ichamatidis · 2021-07-16T17:19:10Z

@NicolasHug Yeah copy-paste error, but the HTTP error persists even with wrong parameters, thus we can narrow that the error happens before any checks

NicoHambauer · 2021-07-17T16:13:32Z

Similar error here!

deepsworld · 2021-07-21T20:43:10Z

Noticed similar issue on personal machine with pytorch:1.9 and torchvision:0.10.0

farleylai · 2021-07-21T22:14:55Z

Likely something to do with pytorch=1.9:

Works with python=3.7 and pytorch=1.7

Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.7.0'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

Works with python3.9 and pytorch=1.7.1 and 1.8.1.

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
import torch
>>> torch.__version__
'1.7.1'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

Failed with python=3.9 and pytorch=1.9

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.9.0'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

mattpopovich · 2021-07-21T22:36:01Z

Similar as @farleylai, no dice with pytorch 1.9.0. This was working previously with pytorch 1.7.0 and 1.8.0. Although, that could be a coincidence if network issues are the problem.

root@pc:~#  python3
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.9.0a0+gitd69c22d'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

pmeier · 2021-07-22T05:13:18Z

No need for more repros. This behavior was introduced in pytorch/pytorch#56138 which was shipped with 1.9.0 and will be fixed by pytorch/pytorch#61761. I don't know if we release a 1.9.1 that includes this fix since it seems to affect quite a few users (@seemethere ?). If not we have to wait for the 1.10 release.

shankhanil007 · 2021-07-22T05:52:57Z

I have a temporary fix for this problem.

" base_model= torch.hub.load('pytorch/vision:v0.9.0', 'densenet121', pretrained=True, progress=True) " : gives me "HTTP Error 403: rate limit exceeded"

BUT,

" base_model = torchvision.models.densenet121(pretrained=True, progress=True) " : works completely fine

pmeier · 2021-07-22T06:05:47Z

Yes, this is a problem with torch.hub. You can access everything through torchvision directly.

oke-aditya · 2021-07-22T06:09:22Z

Yes the hack fix for now is to use

model = torchvision.models.densenet121(pretrained=True, progress=True)

instead of

model = torch.hub.load('pytorch/vision:v0.9.0', 'densenet121', pretrained=True, progress=True)

Note that this hack has a small problem.
torch.hub does not need torchvision to be installed. It will load the model without torchvision being installed through pip / conda.
Whereas the hack to load using torchvision needs torchvision to be installed.

pmeier · 2021-07-22T06:17:46Z

the hack fix

This is not a hack, but a stable feature of torchvision. If you have torchvision installed, one could also say it is superior, because torch.hub downloads the repo in addition to the weights. Still, you are correct, torch.hub is the way to go if you don't have access to torchvision or don't want it as dependency.

For anyone wondering what can be loaded by torch.hub, have a look at

vision/hubconf.py

Lines 1 to 21 in 300a8a4

    
           # Optional list of dependencies required by the package 
        
           dependencies = ['torch'] 
        
           # classification 
        
           from torchvision.models.alexnet import alexnet 
        
           from torchvision.models.densenet import densenet121, densenet169, densenet201, densenet161 
        
           from torchvision.models.inception import inception_v3 
        
           from torchvision.models.resnet import resnet18, resnet34, resnet50, resnet101, resnet152,\ 
        
               resnext50_32x4d, resnext101_32x8d, wide_resnet50_2, wide_resnet101_2 
        
           from torchvision.models.squeezenet import squeezenet1_0, squeezenet1_1 
        
           from torchvision.models.vgg import vgg11, vgg13, vgg16, vgg19, vgg11_bn, vgg13_bn, vgg16_bn, vgg19_bn 
        
           from torchvision.models.googlenet import googlenet 
        
           from torchvision.models.shufflenetv2 import shufflenet_v2_x0_5, shufflenet_v2_x1_0 
        
           from torchvision.models.mobilenetv2 import mobilenet_v2 
        
           from torchvision.models.mobilenetv3 import mobilenet_v3_large, mobilenet_v3_small 
        
           from torchvision.models.mnasnet import mnasnet0_5, mnasnet0_75, mnasnet1_0, \ 
        
               mnasnet1_3 
        
           # segmentation 
        
           from torchvision.models.segmentation import fcn_resnet50, fcn_resnet101, \ 
        
               deeplabv3_resnet50, deeplabv3_resnet101, deeplabv3_mobilenet_v3_large, lraspp_mobilenet_v3_large

mattpopovich · 2021-07-24T05:55:25Z

Temporary fix by @malfet that he mentioned in this comment that fixes things for me:

As a workaround, can you please adding the following line before making any "torch.hub" calls:
torch.hub._validate_not_a_forked_repo=lambda a,b,c: True

dywsjtu · 2021-08-08T09:21:14Z

Temporary fix by @malfet that he mentioned in this comment that fixes things for me:
As a workaround, can you please adding the following line before making any "torch.hub" calls:
torch.hub._validate_not_a_forked_repo=lambda a,b,c: True

That works for me too.

wally456456 · 2021-08-17T17:24:56Z

You can simply download pretrained model by torchvision.model
it seems like there is bug in torch.hub

monajalal · 2021-08-24T02:58:36Z

I was able to reproduce the error on CentOS 7 https://stackoverflow.com/questions/68901236/urllib-error-httperror-http-error-403-rate-limit-exceeded-when-loading-resnet1

AtomicVar · 2021-10-11T04:56:15Z

PR#64677 just added an option skip_validation: bool when loading a model from GitHub. So just add skip_validation=True like

model = torch.hub.load("pytorch/vision", "resnet18", skip_validation=True)

Skip repo validation for TorchHub models due to known bug in `torch.hub` with Github CI (pytorch/vision#4156)

* Added notebook for SlowFast video recognition Notebook demonstrating conversion and inference of the SlowFast model for Action Classification using OpenVino. * Add README for SlowFast Video Recognition notebook Added README describing the objectives, contents and installation steps for the SlowFast video recognition notebook * Update README Added entries for the SlowFast video recognition notebook in notebooks overview tables * Add binder badge for SlowFast video recognition notebook * Styling changes * Fix failed build- missing dependency - Added missing dependency causing build to fail - Updated README * Use proper type classes for type hinting Use classes from the typing library for type hinting generics to ensure backward compatibility with earlier Python versions * Text and styling changes Fix typos and resolve styling issues in the SlowFast Video Recognition notebook. * Potential fix for failing build- skip torch.hub repo validation Skip repo validation for TorchHub models due to known bug in `torch.hub` with Github CI (pytorch/vision#4156) * Removed torch dependency in the data processing pipeline Use numpy instead of pytorch for processing input data to avoid torch dependency during model deployment. * style fix fix missing whitespace around arithmetic operator * Install dependencies in quiet mode * Add link to dataset

y6216886 · 2023-11-30T02:57:40Z

This is because we are using GitHub API to check the existence of the repo. I have submitted a PR to ignore HTTP error 403. To work around this error temporarily, you can delete line 163 on torch/hub.py
_validate_not_a_forked_repo(repo_owner, repo_name, branch)  

it works, the bug is gone.

* pytorch/vision#4156

pmeier added module: hub bug labels Jul 7, 2021

NicolasHug mentioned this issue Jul 7, 2021

No module named 'torchvision.models.mobilenetv2' #4157

Closed

pmeier added the high priority label Jul 8, 2021

pytorch-probot bot added the triage review label Jul 8, 2021

nairbv mentioned this issue Jul 19, 2021

HTTP Error 403 for torch.hub ResNet pytorch/pytorch#61755

Closed

n2cholas mentioned this issue Jul 20, 2021

pretrained_resnet HTTP Error 403: rate limit exceeded n2cholas/jax-resnet#4

Open

AK391 mentioned this issue Aug 12, 2021

resnet18 demo not working. Shows following error message gradio-app/gradio#230

Closed

NicolasHug mentioned this issue Aug 16, 2021

I got an error when loading deeplabv3_resnet101. #4277

Closed

oke-aditya mentioned this issue Aug 18, 2021

Models not pushed to torch hub #4289

Open

6 tasks

pmeier mentioned this issue Aug 24, 2021

Don't validate github url before downloading in hub pytorch/pytorch#61761

Closed

Martynas99 mentioned this issue Sep 2, 2021

HTTP Error in Downloading Models facebookresearch/detr#433

Closed

jamesmcclain mentioned this issue Sep 11, 2021

External Model Definition Functionality Potentially Broken in PyTorch 1.9 azavea/raster-vision#1271

Closed

glenn-jocher mentioned this issue Sep 29, 2021

Revert torch.hub.load() test ultralytics/yolov5#4986

Merged

xhluca mentioned this issue Nov 10, 2021

How can run PyTorch Implementation of AnimeGANv2 on Google colab notebook? bryandlee/animegan2-pytorch#20

Closed

yihong0618 mentioned this issue Nov 21, 2022

ci: make ci stable mli/autocut#52

Merged

rajatkrishna added a commit to rajatkrishna/openvino_notebooks that referenced this issue Mar 17, 2023

Potential fix for failing build- skip torch.hub repo validation

04f8af8

Skip repo validation for TorchHub models due to known bug in `torch.hub` with Github CI (pytorch/vision#4156)

gregbugaj mentioned this issue Aug 25, 2023

Unable to start Executor due to HTTPError 403: 'rate limit exceeded' marieai/marie-ai#78

Open

FreedomXD mentioned this issue Dec 11, 2023

英文文本会自动翻译 mli/autocut#115

Open

19 tasks

yunchu mentioned this issue May 27, 2024

Update for using locally stored pretrained weights in the CI for torch hub models openvinotoolkit/training_extensions#3541

Merged

8 tasks

Maxvgrad pushed a commit to Maxvgrad/perceiver-rnn that referenced this issue Jun 21, 2024

fix: create resnet from torchvision

e0cb0c9

* pytorch/vision#4156

Maxvgrad mentioned this issue Jun 21, 2024

fix: create resnet from torchvision Maxvgrad/perceiver-rnn#6

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP Error 403: rate limit exceeded when loading model #4156

HTTP Error 403: rate limit exceeded when loading model #4156

ichamatidis commented Jul 6, 2021 •

edited

Loading

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 6, 2021

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 6, 2021

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 7, 2021 •

edited

Loading

ptrblck commented Jul 7, 2021

pmeier commented Jul 7, 2021 •

edited

Loading

ailzhang commented Jul 7, 2021

AK391 commented Jul 7, 2021

Vernerrf commented Jul 8, 2021

pmeier commented Jul 8, 2021

SomTambe commented Jul 15, 2021

cyyever commented Jul 16, 2021 •

edited

Loading

pmeier commented Jul 16, 2021

NicolasHug commented Jul 16, 2021

ichamatidis commented Jul 16, 2021

NicoHambauer commented Jul 17, 2021

deepsworld commented Jul 21, 2021 •

edited

Loading

farleylai commented Jul 21, 2021 •

edited

Loading

mattpopovich commented Jul 21, 2021 •

edited

Loading

pmeier commented Jul 22, 2021

shankhanil007 commented Jul 22, 2021

pmeier commented Jul 22, 2021

oke-aditya commented Jul 22, 2021

pmeier commented Jul 22, 2021

mattpopovich commented Jul 24, 2021 •

edited

Loading

dywsjtu commented Aug 8, 2021

wally456456 commented Aug 17, 2021

monajalal commented Aug 24, 2021

AtomicVar commented Oct 11, 2021 •

edited

Loading

y6216886 commented Nov 30, 2023

HTTP Error 403: rate limit exceeded when loading model #4156

HTTP Error 403: rate limit exceeded when loading model #4156

Comments

ichamatidis commented Jul 6, 2021 • edited Loading

🐛 Bug

To Reproduce

Expected behavior

Environment

Error Output

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 6, 2021

Error Output

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 6, 2021

oke-aditya commented Jul 6, 2021

ichamatidis commented Jul 7, 2021 • edited Loading

ptrblck commented Jul 7, 2021

pmeier commented Jul 7, 2021 • edited Loading

ailzhang commented Jul 7, 2021

AK391 commented Jul 7, 2021

Vernerrf commented Jul 8, 2021

pmeier commented Jul 8, 2021

SomTambe commented Jul 15, 2021

cyyever commented Jul 16, 2021 • edited Loading

pmeier commented Jul 16, 2021

NicolasHug commented Jul 16, 2021

ichamatidis commented Jul 16, 2021

NicoHambauer commented Jul 17, 2021

deepsworld commented Jul 21, 2021 • edited Loading

farleylai commented Jul 21, 2021 • edited Loading

mattpopovich commented Jul 21, 2021 • edited Loading

pmeier commented Jul 22, 2021

shankhanil007 commented Jul 22, 2021

pmeier commented Jul 22, 2021

oke-aditya commented Jul 22, 2021

pmeier commented Jul 22, 2021

mattpopovich commented Jul 24, 2021 • edited Loading

dywsjtu commented Aug 8, 2021

wally456456 commented Aug 17, 2021

monajalal commented Aug 24, 2021

AtomicVar commented Oct 11, 2021 • edited Loading

y6216886 commented Nov 30, 2023

ichamatidis commented Jul 6, 2021 •

edited

Loading

ichamatidis commented Jul 7, 2021 •

edited

Loading

pmeier commented Jul 7, 2021 •

edited

Loading

cyyever commented Jul 16, 2021 •

edited

Loading

deepsworld commented Jul 21, 2021 •

edited

Loading

farleylai commented Jul 21, 2021 •

edited

Loading

mattpopovich commented Jul 21, 2021 •

edited

Loading

mattpopovich commented Jul 24, 2021 •

edited

Loading

AtomicVar commented Oct 11, 2021 •

edited

Loading