Hit Docker rate limits?

[ParadingLunatic]

Unfortunately it seems like using this hosted in a docker on my system caused my IP to get flagged for a rate limit. Now when I go to hub.docker.com and I'm not logged in, searches come back with no results and if I try to go directly to a repository I know exists, I get a 404. It only happens when I'm 1) not logged in and 2) only from my IP. I tested this from my cell phone while connected to wifi and then after disconnecting from wifi. Trying to reach a known repository while connected to wifi resulted in a 404. I disconnected the wifi and went to the same URL and it worked fine.

I have 11 different RSS feeds I use to check for new images and I believe my RSS server refreshes every 15 minutes. Guess that was too much for Dockers liking.

[ParadingLunatic]

Go figure, right after sending this, it appears the rate limit has been lifted for my IP. Was happening for about 24 hours, so I'm guessing that's how long it's required to be reset.

[TheConnMan]

Interesting, I have not seen my hosted version of this get flagged due to rate limiting. What action should we take for this issue, anything?

[ParadingLunatic]

Honestly I have no clue. I was pretty shocked when it happened considering what I've read just regular queries would require tons of hits in order to get rate limited. I can't imagine my 11 feeds every 15 mins was enough to trigger it but who knows. Only reason I think it was rate limiting was because prior to getting the 404's when I went directly to a repository, I got a "too many requests" page. Eventually it turned into a 404, even though I know the repository existed. The only thing I can think of is maybe provide a way to pass your docker account login across or possibly use an access token. Not sure how much that'll help though.

Also to provide some other info. I only do maybe 10-15 pulls tops on some days, which is far below the limit for pulls, so I know it wasn't that either. Was just surprising seeing that it was failing, only to find out it was directly associated to my IP.

[TheConnMan]

Yeah, I've never run into that during local dev where I've made hundreds of calls in a day.

[khassel]

I ran also into this problem.

It depends on which docker repository you are querying, e.g. this one has more than 10000 tags.

The vercel app cannot retrieve the tags, may due to memory restrictions.

This project queries all tags and with the default of 100 tags per page you need more than 100 api calls only for 1 repo.

May one idea would be to restrict the tag number per default to 100 putting a checkbox in the web-ui called only get 100 tags which is checked by default.

[TheConnMan]

Yikes. Yeah, someone else ran into a timeout on PHP which leads me to think we need to limit the default API to x number of pages of images. Not sure what x should be, though.

[ParadingLunatic]

Well, this hasn't happened in a while....but it happened again. Getting errors trying to pull the info down and when I go to hub.docker.com I get "Too many requests". Just went to your public instance, looks like you may be experiencing the same?

[ParadingLunatic]

Interesting. Stopped the container for a bit and after several minutes I was able to browse hub.docker.com. Even do searches. Started the container and started having issues again.

[TheConnMan]

If you were hit both locally and on the public instance I expect it was a Docker Hub incident. I have had issues with my public instance, but they're hosting-related. Local issues at the same time mean Docker Hub is the common denominator.

[TheGreatestJannet]

I seem to be hitting the same issue. I can't search docker hub and the docker-hub-rss doesn't return any information which trying to setup a feed. Using Tor searches work fine so my IP must be blocked. I am the only user of the instance.

[p-schneider]

I seem to be hitting rate limits for two weeks now.
(So I stopped the container and only occasionally start it to check if it started working again.)
The public instance at https://docker-hub-rss-theconnman.vercel.app/ also seems to be unable to fetch anything.
It seems like #37 could fix the current issues.

[TheConnMan]

Hey folks, thanks for your patience on this. I've merged #37 and deployed it to prod. I'll need to upload the Docker image manually as Travis is no longer active.

Thank you @visit1985 for the fix!

[TheConnMan]

I've uploaded all Docker images (latest, latest-dev, 0.4.3, and 0.4.4) to Docker Hub. I'll need to figure out how to push the arm images up without Travis. Given that this fix is out in master, I will close this ticket. Thanks again for everyone's patience and interest in the project!