The buildbot server gets tons of HTTP requests

[vstinner]

When I watch tail -f master/http.log, it's hard for me to follow because they are some many HTTP requests. It seems like most of them are coming from crawler bots. For example, in a single second, the server got 50 requests:

"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /plugins/wsgi_dashboards/styles.css HTTP/1.1" 404 145 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /plugins/grid_view/styles.css HTTP/1.1" 404 145 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651573/logs HTTP/1.1" 200 155 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651599/logs HTTP/1.1" 200 155 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651607/logs HTTP/1.1" 200 153 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651606/logs HTTP/1.1" 200 60 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651574/logs HTTP/1.1" 200 193 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651572/logs HTTP/1.1" 200 155 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651598/logs HTTP/1.1" 200 155 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651349/logs HTTP/1.1" 200 60 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/steps/19651350/logs HTTP/1.1" 200 60 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /plugins/grid_view/scripts.js HTTP/1.1" 200 21412 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /plugins/console_view/scripts.js HTTP/1.1" 200 53955 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builds?builderid=2068&number__eq=91&number__eq=92&number__eq=93 HTTP/1.1" 200 314 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/buildrequests/3402886 HTTP/1.1" 200 212 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builds/2399387/steps HTTP/1.1" 200 528 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builds/2399387/properties HTTP/1.1" 200 479 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning-list.js HTTP/1.1" 200 736 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning-list.js HTTP/1.1" 200 736 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.css HTTP/1.1" 200 1411 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.js HTTP/1.1" 200 28827 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.js HTTP/1.1" 200 28827 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.css HTTP/1.1" 200 1411 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/buildsets?rebuilt_buildid=2399387&limit=1&order=-bsid HTTP/1.1" 200 65 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/workers/39 HTTP/1.1" 200 556 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-CQKNRskk.css HTTP/1.1" 200 212277 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-CQKNRskk.css HTTP/1.1" 200 212277 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.css HTTP/1.1" 200 1411 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/buildsets?rebuilt_buildid=2414161&bsid__gt=50747&limit=1&order=-bsid HTTP/1.1" 200 65 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builds?builderid=448&number__eq=7695&number__eq=7696&number__eq=7697 HTTP/1.1" 200 301 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning-list.js HTTP/1.1" 200 736 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.js HTTP/1.1" 200 28827 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/buildsets?rebuilt_buildid=2417504&bsid__gt=50820&limit=1&order=-bsid HTTP/1.1" 200 65 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-CQKNRskk.css HTTP/1.1" 200 212277 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/buildsets/36957 HTTP/1.1" 200 372 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-D1Qujclx.js HTTP/1.1" 200 1123342 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-D1Qujclx.js HTTP/1.1" 200 1123342 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /assets/index-D1Qujclx.js HTTP/1.1" 200 1123342 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET / HTTP/1.1" 200 2351 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builders/1417 HTTP/1.1" 200 213 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builds/2423022/properties HTTP/1.1" 200 511 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /api/v2/builders/2068 HTTP/1.1" 200 207 "https://buildbot.python.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning-list.js HTTP/1.1" 200 736 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.css HTTP/1.1" 200 1411 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
"127.0.0.1" - - [03/Jul/2026:13:04:21 +0000] "GET /browser-warning.js HTTP/1.1" 200 28827 "https://buildbot.python.org/all/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"

The HTTP user agent looks like legit users, but I don't think that there are so many people using buildbot.python.org at the same time.

According to htop, the CPU usage of the buildbot process is around 100% most of the time.

---

Sometimes, showing logs of a build fails with a timeout. For example, go to https://buildbot.python.org/#/builders/1645/builds/3013 unfold "test": the logs are empty whereas it should display the ~10 last lines, and if I click on [Raw], it fails with a timeout:

504 Gateway Time-out
The server didn't respond in time.

Maybe the high rate of HTTP request explains why the server times out?

[JacobCoffee]

Can close if/when the requests settle, not convinced python/psf-salt#644 will be enough

[vstinner]

I checked http.log: we got 34973 requests between 16:09:48 and 16:26:56 (1028 seconds): so we got 34.0 requests/second in average.

• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36: 6232 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36: 6062 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36: 4321 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36: 4131 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36: 3094 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36: 2642 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36: 2568 requests
• Lightpanda/1.0: 1507 requests
• -: 977 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36: 613 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36: 590 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0: 583 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0: 572 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36: 423 requests
• Sogou web spider/4.0: 304 requests
• Datadog/Synthetics: 153 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.33 Safari/537.36: 125 requests
• GitHub-Hookshot/227665c: 33 requests
• python-requests/2.32.5: 17 requests
• Muffet/2.11.1: 11 requests
• curl/8.5.0: 4 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36: 2 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36: 2 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36: 1 requests
• Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/172.1: 1 requests
• Mozilla/5.0 (Macintosh; Intel Mac OS X 13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/260.1: 1 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36: 1 requests
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36: 1 requests
• Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412.7 (KHTML, like Gecko) Safari/412.6: 1 requests
• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36: 1 requests

Total: 30 user agents and 34973 requests.

I expected Lightpanda/1.0 user agent to be blocked, but I still see in buildbot HTTP logs.