dockerhub containers not up-to-date with registry

[osallou]

seems details are not fetched for all containers.... (example seaborn...)

logs show:

...
INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- pandas
INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- seaborn
INFO:biocontainers.dockerhub.models:tool retrieved from DockerHub -- biocontainer-pipelines

then fail on other pages

Traceback (most recent call last):
File "pipelines.py", line 268, in
main()
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "pipelines.py", line 234, in main
import_dockerhub_containers(config, config_profile)
File "pipelines.py", line 63, in import_dockerhub_containers
dockerhub_containers = reader.get_containers()
File "/biocontainers-backend/biocontainers/dockerhub/models.py", line 114, in get_containers
response = call_api(url)
File "/usr/local/lib/python3.7/site-packages/ratelimit/decorators.py", line 112, in wrapper
return func(*args, **kargs)
File "/usr/local/lib/python3.7/site-packages/ratelimit/decorators.py", line 80, in wrapper
return func(*args, **kargs)
File "/biocontainers-backend/biocontainers/common/utils.py", line 13, in call_api
raise Exception('API response: {}'.format(response.status_code))
Exception: API response: 429

[ypriverol]

I will take care of it.

[osallou]

429 Too Many Requests
The user has sent too many requests in a given amount of time ("rate limiting").

[osallou]

seems we need to temporize, but we have to see dockerhub api limits

[osallou]

log says that sending report to slack, you should see errors somewhere

[ypriverol]

Yes, we have the issue there.

[osallou]

https://docs.docker.com/docker-hub/api/latest/#tag/resources

If you have hit the limit, you will receive a response status of 429 and the X-Retry-After header in the response.

The X-Retry-After header is a unix timestamp of when you can call the API again.

*Note: These rate limits are separate from anti-abuse and Docker Hub

download, or pull rate limiting.

[osallou]

sample api call via curl shows:

 x-ratelimit-limit: 180
 x-ratelimit-reset: 1683218176
 x-ratelimit-remaining: 180

[mboudet]

@ypriverol Is this issue still active? Newly merged containers do not seems to be in the registry.
(Ex: BioContainers/containers#550)