Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement download endpoint limits #3651

Closed
8 tasks done
patphongs opened this issue Mar 27, 2019 · 1 comment · Fixed by #3695
Closed
8 tasks done

Implement download endpoint limits #3651

patphongs opened this issue Mar 27, 2019 · 1 comment · Fixed by #3695

Comments

@patphongs
Copy link
Member

patphongs commented Mar 27, 2019

In order to rate limit the amount of download requests per IP, we need to restrict the downloads endpoint to only 1 download key. Then we will be able to apply rate limits to that key as desired. This will protect us from a single IP requesting too many downloads all within a short timeframe.

Rate limit downloads

Completion criteria:

  • Check IPs for internal network traffic to see if rate limits will interfere with internal downloads
    For wired and wireless public IP's for FEC HQ (see #security-practices channel for the IPs) and the most download requests we got was 20/hour. One anomaly was load testing on 4/1 - those can been discarded. After finding this, I think we should target 60 requests/hour/IP, which is triple our busiest internal day in the past 90 days (logs: https://logs.fr.cloud.gov/goto/433b8ffe039c6dcbb885bb3b235ea74a
  • Create new download key, one per environment. CMS should only use this key for downloads. CMS dev and feature should share a key
  • Rate limit the new key. Each ping to check task status is considered a request. We may want to start with allowing 100 requests per hour per IP. Calculations are as follows: 100 requests / 4 average pings per download request = ~25 CSV download files can be requested per hour per IP
  • Create API restrictions on the /download/ endpoint to only whitelist requests from new download keys
  • Save env vars in stage and prod environments for DOWNLOAD_WHITELIST_API_KEY_ID (save key ID and not key name). (dev doesn't use the API umbrella)
  • Create a follow-up front-end ticket for error handling and update the download key in the CMS. On the front end, add error handling messages for exceeding the request limit. Ex: "You have exceeded the maximum allowed download requests this hour". Handle over rate limit message for downloads fec-cms#2842
  • Make a ticket to unblock particular IPs from downloads (included in the PR as env var)
  • Turn on downloads protection with RESTRICT_DOWNLOADS=True in stage (We will do prod after the deploy)
@patphongs
Copy link
Member Author

Make a feature flag for this in the API

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants