Repository navigation
IMPORTANT: Rate-limiting the Tectonic cloud service [input requested] #1391
Replies: 11 comments 3 replies
|
Here's a little update: Most importantly, I thought that it was maybe already helping, but I realized that yesterday's rate limit wasn't actually being applied! I needed to explicitly turn the Azure WAF rule into "prevention mode". I have now done this. I've also added a more specific rule that applies a quite strict rate limit to requesting any It appears that these rate limits are enforced by individual Front Door servers, because while I've confirmed that they're in effect, I can download the index file around a dozen times before actually encountering the rate limit rejection. I presume that my requests are rotating between servers. In support of the above, I had started gathering detailed traffic logs to better understand what's going on. In the past 24 hr, that's about 5 million requests, and about 10 GB of log data. Admittedly, Azure's log format is ridiculously bloated.) It remains true that I don't see any one single culprit in terms of client IP addresses. The busiest class-C IPv4 address space (over the past 24 hr) for general requests is at about 3% of all requests. The busiest IP address for requesting the bundle index file is about 10% of all requests. Nuking those would do something, but not much compared to an overall tripling of traffic over the past 90 days. The busiest client IP addresses seem to be predominantly associated with large data centers. That's not surprising. Analysis of User-Agents isn't too helpful. The default Tectonic client leaves this blank and that is what I see for the vast, vast majority of traffic. There are requests with interesting user-agents like "Codex-Tectonic-Range-Proxy/1.0" and "Codex-LaTeX-Compiler/1.0" that may be associated with LLM code or agents, but they're a tiny fraction. I'm motivated to update the Tectonic HTTP client to start filling in the User-Agent here to at least provide a bit more information, but the takeaway is that the vast majority of traffic is coming from the Tectonic Rust code, not alternative implementations. That's good to confirm, but also not surprising. Given all the above, I don't see any obvious thing to do besides enforce blanket rate limits. If many requests were coming from unusual IPs or user-agents, I could block those specifically; but the traffic is very broad-based and I see no clear way to distinguish normal users from whatever is driving up the usage dramatically. The next tier of control would be, like, issuing API keys, but that would be a huge huge increment in complexity and an enormous blow to convenience. Now that I'm actually enforcing the rate limits, I'll keep an eye on things and see how they look. The WAF logs are showing that the limits are indeed kicking in for various clients (now that I've actually turned them on ...), and hopefully I'll see that reflected in the traffic stats over the next few days. |
|
Today's update: no appreciable decrease in traffic over the past 24 hours, so I'm going to decrease the rate limits. As things currently stand, I just don't see any other option for near-term mitigation. My top concern is if the rate limits start limiting people who are building documents as individuals with "normal" usage patterns. If you're using Tectonic and encounter a failure that appears to be due to rate limiting, please please post here — there are probably hundreds of people in the same situation, but from experience I can say that only a tiny fraction will be in a position to report the problem. If you're reading this, you're part of that fraction. I've also tried full-on blocking a list of specific IP addresses that are repeatedly hitting the index-file rate limit. I'm reluctant to ban specific IPs, but only a small number of addresses are responsible for the vast majority of these rate-limit triggers, and the Azure rate limits expire after no more than 5 minutes. So a full-on block has the potential to make much more of a difference than any rate limit I can impose. |
|
Today's update: I was offline this weekend but now I'm back and I've reviewed logs from the past few days. It appears that blocking specific IPs is very effective, so I've gone ahead and blocked a few more — there are currently 67 in total. It looks like this may be enough to get the usage under control, but I'll need to see how things evolve. This may imply that I'll need to manage an IP blocklist in perpetuity, which wouldn't be my favorite thing, but a small price to pay if it keeps the service broadly available to small-scale users. |
|
The bots (?) are striking back: 
This is showing a surge in traffic starting about 12 hours ago. In response I've implemented a much more aggressive set of IP blocks, derived automatically from the firewall rate-limit logs. This currently bans ~2,100 distinct IP addresses. We'll see if it helps. As I've been working on this, I've been doing some informal research about how people are handling bot traffic these days. It is widely-agreed that bot traffic is becoming much harder to handle because crawlers are getting much better at spreading their traffic around lots of IP addresses. Unfortunately, the traffic to the Tectonic servers in particular is extremely "vanilla" and so most of the current techniques for blocking bots, which seem to be deemphasizing simple IP-based blocking, seem unlikely to be able to work here. The other widely-agreed-upon opinion is that proof-of-work schemes like Anubis work well. This comes up because when people are talking about bot traffic, they're almost always doing so in the context of interactive web traffic, where you're trying to allow actual humans to view your website in a browser. In contrast, the service here is basically an API, and normally these are protected by some kind of user registration and/or API key framework. I explicitly chose not to set up anything like that for Tectonic out of a combination of laziness and desire for openness. Coming from that standpoint, the proof-of-work model is interesting to think about — there's no reason you couldn't use it in an API context. And it would require special support in the client, but so would the adoption of an API key system. That being said, currently I would guess that proof-of-work wouldn't work well here because the nature of the incoming traffic suggests that people are actually running Tectonic-specific client code, in which case any proof-of-work requirement that isn't absolutely awful for actual human users will probably fail to stop robotic users as well — I don't think this traffic is coming from crawlers that are trying to index billions of pages. I also set up a |
Question: CDN Alternatives?After a hair-raising 24 hours in which the Tectonic servers' outgoing data rate was higher than it's ever been, despite blocking thousands of IPs, traffic has subsided a great deal. Hopefully that will continue to be the case. But I'm starting to feel that Azure FrontDoor may not be the best choice for this service. I was wondering if people had any input about the following alternatives:
For context, I'm currently looking at (pessimistically) ~25 TB of traffic per month and around 150 million requests. This service needs almost nothing clever, but I would like some capability to block specific IP addresses and ASNs. I have looked into blocking by country geolocation, but this seems just too coarse-grained — I would have to block all of China and Germany to make a dent in the problematic traffic. According to the AWS docs, the Pro tier would be able to do this with a lot of headroom for traffic at just $15/month. Compared to what I'm paying now, this seems too good to be true. It looks like the Pro tier would also have all of the WAF rule functionality I expect to need. The bunny.net volume tier would be more like $125/month, which would still be an incredible improvement over the current situation. On paper I can't see any specific way in which Bunny.net would definitely be better than AWS, but I know that Bunny is aimed at the cheap-no-frills segment of the market, which is exactly where I land. Given that, it's a little suspicious to me that the AWS flat-rate plan seems to be such a better deal. update: Aha! AWS Pro only allows 10M requests/month, so I would probably have to bump up to the Business tier, unless the traffic blocking is much more effective. Azure doesn't support ASN blocking in the basic tier, and unless the headline prices are really underselling things, it's wildly expensive compared to these options. |
|
I've set up a trial experiment of using bunny.net as the Tectonic CDN. I've done this by creating a new subdomain, Compared to Azure, Bunny.net is letting me configure general download bandwidth rules, and I can block up to 1000 ASNs. I haven't turned on ASN blocking yet so as to gather data first, but the bandwidth limits might result in a behavior change. It feels like the initial cache fill is a lot slower, but this is a one-time cost for typical use cases, so I am extremely OK with that. Totally unrelated, I was reluctant to try Cloudflare as a CDN option, but I looked into it because it seemed like the Free plan might cover Tectonic's needs, and free is pretty compelling. But it doesn't work because Cloudflare's non-Enterprise plans will only cache a maximum asset size of 512 MB, and the underlying Tectonic bundle files are several gigs. Even though the individual requests coming in only ask for small byte ranges within the overall large files, Cloudflare wants to cache the whole underlying file, which it then refuses to do. So without an enterprise subscription, Cloudflare's cache hit rate would be miniscule. Oh well. |
|
The bunny.net experiment is going well from my perspective. The traffic level is still absurdly high, but costs are acceptable, and it seems to me that ASN-level blocking is working decently. I have heard no complaints about blocks, at least. I will plan to fully switch over the Tectonic data services to bunny.net in the coming days, unless some serious issue emerges. edit: Spoke too soon — there are at least a few issues filed relating to the new CDN. I will dig into these before committing fully to switching over. |
|
Cloudflare R2 might be worth a look. It's object storage rather than the CDN you tried, and egress is free. Should work out to around $20–50/month at your volumes even without caching. There's also Project Alexandria for which Tectonic should be a great candidate. It includes Enterprise plan upgrades which would raise the 512 MB base cache limit to 5 GB. |
|
Tonight I'm going to start trying out Cloudflare with R2 as the "data1b" alternative. I'll need to switch the DNS for fullyjustified.net over to Cloudflare (required by the Free plan that I'm starting with), which is an inconvenience — but only a minor one, hopefully. Thanks for the pointer to Project Alexandria @ogoregen — that could be helpful. For my day job I've become aware of their Project Galileo, but I didn't realize that there were multiple Projects ongoing! |
|
OK, the Cloudflare experiment has been going very well, as far as I'm concerned. I feel a little bad that the CDN cache hit rate is ~zero due to the file-size issues mentioned above, but even given that, the running cost is coming in at less than $2 / day, with no IP blocks or rate limits applied. If anyone is having issues with the data distribution from the "data1b" endpoint as it currently stands, voice them soon. If no showstoppers emerge I will plan to complete the switchover and decommission the Azure and Bunny.net CDN services. |
|
Today I found out that tectonic has been bundled within the official ChatGPT (codex) app! On macOS it is placed under: And it's also bundled in the Linux .deb version (not sure about the windows version). I think this must have contributed to the surge, and it's reasonable to ask OpenAI for help with hosting. They are rich people! |
|
I strongly suspected this — thanks for confirming! In the magical world of AI apparently all web services are free to use at whatever scale you want (we are also seeing this in my day job), and the rest of us just have to deal with the consequences. |
|
Yes indeed... On the other hand I think this shows the great value of tectonic (as an alternative to other tex engines/distributions) and it could be a good opportunity to ask for industry support. It would be nice if openai could fund this project 😆 |
|
Have you talked to OpenAI about this? The new ChatGPT (codex) app on macos has a new LaTeX side panel and it automatically compiles .tex documents by default, with tectonic! I can imagine this will add even more pressure to the server... 
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I have a question for the hive mind in the most recent post — please scroll to the bottom!
Hi folks —
It's been a while! Unfortunately, while I would love to say that I've finally been able to start carving out time to work on Tectonic again, I have to chime in with a somewhat less fun update.
The Tectonic cloud service runs on Microsoft Azure, and I pay for it out of pocket. While my server bills have been creeping up slowly over time, they've skyrocketed over the past few months. Unfortunately, from what I'm able to see so far, there's no one simple cause of this, so I've applied a rate limit to the cloud service in an attempt to keep things under control. I have no idea whether (1) this will help at all or (2) what kinds of effect it will have on the user experience, but I need to do something. Hopefully it won't affect day-to-day usage perceptibly, but I just don't know.
Here are the key plots from the Azure Front Door (CDN) analytics from the past 90 days:
Over the past three months, the Tectonic CDN has served up more than 34 TB of data over almost 200 million requests. In one way that is very cool! But it's not free.
The difference between the two lines in the top panel indicates that the CDN is working fantastically well: the caching is extremely effective and the origin server is doing OK. The average request size is about 50kb, which seems reasonable. The logs also tell me that the requests are virtually all going to the main bundle file. So at this (fairly superficial) level of analysis, the traffic all looks reasonable ... just that there is an absolutely enormous amount of it. (At ~2 million requests per day and ~350 files needed to build a document for the first time, that's very handwavily a fresh build every 10 seconds, 24/7.)
You will also see that the traffic has been growing steadily for the past few months. With the data that I currently have available, I can't really tell why that is. @CraftSpider has gotten some new releases out recently (yayyyyy!!!!!!!!!) which at least raises the possibility that maybe there's some kind of new caching bug in the most recent code, but I think it would be have to blazingly obvious/bad to lead to this much more traffic.
Repeated builds of documents by humans should need to hit the bundle only very rarely, so my first guess is that this is something like CI jobs that aren't caching the bundle data. That still leaves the mystery of why usage would start going up so much this summer, though.
Based on my experience with all sorts of other online communities recently, I'm also tempted to blame AI agents — they seem to be driving a major increase in traffic to everything, everywhere, all across the board. But I'm going purely on vibes there.
Regardless, some asks for the community:
All reactions