bug 1199166 - block_user_agents decorator for views

[groovecoder]

Not sure this settings.BLOCKABLE_USER_AGENTS approach is the best one. Need some r? on this overall approach ...

[jezdez]

Those three lines can be written as blockable_user_agents = getattr(settings, 'BLOCKABLE_USER_AGENTS', [])

[groovecoder]

Updated.

[groovecoder]

Oh, still needs tests ...

[groovecoder]

Adding tests with real user agent values showed me that we need to use pattern.search to catch the user agent strings. This is ready for final review & merge.

[groovecoder]

@jwhitlock - can you take this one now?

[jwhitlock]

Is kumascript blocked by this? I'm pretty sure document.get_children and document.as_json are used by KS macros.

[groovecoder]

Not likely but it could potentially block KumaScript if it does something crazy like send 'curl' or 'get' as the user agent string. I'll verify before merging & deploying. (I'll also add YandexBot to the blockable agents)

[jwhitlock]

Should we add https://developer.mozilla.org/en-US/docs/all and https://developer.mozilla.org/en-US/docs/tag/* to robots.txt as well?

[jwhitlock]

(I didn't miss this one - it's in robots.txt already)

[groovecoder]

😛

[jwhitlock]

+r on code, with possible additions to robots.txt

[jwhitlock]

+r on robots.txt additions. Remember to confirm kumascript before it goes to production.

[groovecoder]

• Added the Disallow entries to robots.txt.
• Added YandexBot to BLOCKABLE_USER_AGENTS since it triggered a db-heavy transaction yesterday.

Yup; still need to verify the kumascript user agent sent to Apache/django.

[groovecoder]

After looking thru access logs for other crawling user agents, I also added:

"Baiduspider",
"CCBot",
"ScoutJet",

[groovecoder]

I verified locally that KumaScript requests are sent to Apache without a blocked user agent string:

developer-local.allizom.org:443 127.0.0.1 - - [03/Sep/2015:16:57:06 +0000] "GET /en-US/docs/en-US/Template:dekiscript:page?raw=1 HTTP/1.1" 404 3947 "-" "-"

And cyliang sees the internal requests to production Apache are sent similarly - i.e., with "-" user agent:

 - - [03/Sep/2015:09:30:04 -0700] "GET /en-US/docs/MDN/Kuma/API?raw=1 HTTP/1.1" 200 6722 "-" "-"

So, this is good for merge + deploy.