Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: throttle requests to rate limited domains #97

Merged
merged 12 commits into from
Feb 6, 2023
Merged

Conversation

rafaelcr
Copy link
Collaborator

@rafaelcr rafaelcr commented Feb 6, 2023

  • Save domains that respond with 429 status codes
  • Parse the Retry-After header and save that date for retries, default to 1 hour if not specified
  • Change all timestamp columns to timestamptz to correctly use and compare pg timestamps
  • Upgrade undici to include new error types from fix(types): update error type definitions nodejs/undici#1888

Fixes #86

@rafaelcr rafaelcr linked an issue Feb 6, 2023 that may be closed by this pull request
Copy link
Member

@zone117x zone117x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rafaelcr rafaelcr merged commit 5b75060 into beta Feb 6, 2023
@rafaelcr rafaelcr deleted the feat/429-limits branch February 6, 2023 18:22
blockstack-devops pushed a commit that referenced this pull request Feb 6, 2023
## [0.1.0-beta.6](v0.1.0-beta.5...v0.1.0-beta.6) (2023-02-06)

### Features

* throttle requests to rate limited domains ([#97](#97)) ([5b75060](5b75060))
@blockstack-devops
Copy link

🎉 This PR is included in version 0.1.0-beta.6 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

blockstack-devops pushed a commit that referenced this pull request Feb 22, 2023
## [0.1.0](v0.0.1...v0.1.0) (2023-02-22)

### Features

* add server version to status endpoint and rendered docs ([#76](#76)) ([ba2f7de](ba2f7de))
* enable run modes ([#116](#116)) ([c7a9c55](c7a9c55))
* import sip-019 notifications during boot ([#81](#81)) ([6c28037](6c28037))
* refresh dynamic metadata tokens periodically ([#64](#64)) ([e1c0882](e1c0882))
* throttle requests to rate limited domains ([#97](#97)) ([5b75060](5b75060))

### Bug Fixes

* add `/metadata/v1` prefix to all routes ([#100](#100)) ([a11d4be](a11d4be))
* add cache-control, remove cache for error responses ([#114](#114)) ([e03caf8](e03caf8))
* contract log queries ([4bd2812](4bd2812))
* display cached image in metadata responses ([#104](#104)) ([156e9e2](156e9e2))
* dockerfile CMD path ([#91](#91)) ([de60556](de60556))
* enclose response etag in double quotes ([#113](#113)) ([2b77cfe](2b77cfe))
* etag cache calculation with url prefix ([#111](#111)) ([f872f93](f872f93))
* follow redirects when fetching metadata ([#109](#109)) ([0ab2fbb](0ab2fbb))
* generate git info on docker build ([#93](#93)) ([9808b47](9808b47))
* handle pg disconnections and transaction management ([#92](#92)) ([201d813](201d813))
* ignore invalid ssl certs for metadata fetch ([#107](#107)) ([46e184c](46e184c))
* ignore ts type maps when migrating ([#95](#95)) ([b92b9d8](b92b9d8))
* improve SIGINT handling for queued jobs ([e16fcd5](e16fcd5))
* jsonb type interpretation on endpoints ([5985c80](5985c80))
* jsonb value insertions ([8dff8a6](8dff8a6))
* make importer wait for API height to catch up if it's behind ([#101](#101)) ([930cce3](930cce3))
* manage additional timeout errors on metadata fetch ([e658e1d](e658e1d))
* move from fetch to request to fix ENOBUFS ([9b26439](9b26439))
* persist http agent for metadata fetches ([a30641a](a30641a))
* retry 429 and gateway timeouts ([08cdce6](08cdce6))
* sft_mint detection ([53673b2](53673b2))
* shut down queue at the end of sequence ([e268c79](e268c79))
* support JSON5 metadata strings ([#106](#106)) ([d19634f](d19634f))
* uintcv creation ([780b160](780b160))
* update_at jobs on status or retry change ([c16025f](c16025f))
@blockstack-devops
Copy link

🎉 This PR is included in version 0.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

if a domain responds with 429 on metadata fetch, avoid subsequent calls
3 participants