Repository navigation
Real‐world usage
TPS was originally built to solve a single real-world problem: our digital exhibit platform was wrecked by bot traffic (50% uptime on a good day). Misbehaving bots rotated their IP addresses per request, ignored our sitemap, ignored robots.txt, etc.
Once we had 20 or so bots each making several requests per minute to our most expensive endpoints (search and facets), the stack just couldn't keep up. It was hosted on a shared setup with fairly low resources because it wasn't expected to see insane levels of traffic.
Building TPS and putting it in front of search and facet requests solved the resource problems on day 1. Bots still get to crawl our resources, just not our search pages. Our site stays up. Win-win.
Take a look at our Digital Exhibits Github project for details of how we used TPS to basically save a real application.
TPS has also been deployed to protect our largest collection, Historic Oregon Newspapers. Uptime was never as big an issue as our exhibits platform, but the server's base resource use was very high, and for a short time we had to monitor traffic, do reverse-lookups on IP addresses, and block entire ISPs at the ASN level. Even with millions of IPs blocked, a surge in traffic, or even just an expensive backend operation (loading a batch of new newspaper content) occasionally caused critical services to crash and not recover.
We had to be a lot more deliberate here: unlike our exhibits platform, HON doesn't have a single chokepoint, and HON's content is so big there's always constant "good bot" activity: it takes Google 3-6 months to do a full site crawl at friendly rates (say 2 requests a second). So we set up TPS to allow as much harvesting as we can afford. Bots that identify themselves, stick to our sitemap, and don't DoS us (harvesting agents being run without throttling) can freely harvest whatever they like.
With the TPS report, we're now seeing that in any given day there are 400,000 to four million requests for protected resources. Typically very few render JS, but occasionally we get a flood of smart bots where 20-30% of requests are rendering JS. Interestingly, the number of successful challenges tends to be fairly consistent whether we seen 400k or 4m requests, so we can say with confidence that the vast majority of our traffic is bots.
During major attacks (malicious or not, bot traffic like this is an attack), we have seen as many as 600,000 requests in a single hour, sometimes at levels like that for several hours. (Note to bots: y'all need to calm down. This is a website that has a single state's newspapers, not Wikipedia.)
TPS has kept our site more stable than when we had 1/10th this much traffic, and we block fewer bots. Server load is dramatically lower, we no longer try to chase ASN-level blocking, and responsiveness hasn't been this good in years.