Support bots and phones #52
Conversation
Even the feature phones with WAP...
This can be desktop, mobile or bot. There is some ambiguity with phablets, tablets, netbooks, and game consoles obviously. There should be no such confusion about bots...
|
Ok, I added a kind, which makes it possible for users to broadly categorize traffic without worrying whatever MAUI or PhantomJS is. This is really important for filtering robots from your statistics. Mobile is less essential and the meaning of mobile is a bit blury these days. It could be used to display the mobile version of a website, but that would be slightly wrong. |
|
Thanks much @pepijndevos Bot detection is major enhancement :) tests are failing python setup.py test
# or python tests.pyI am sure above are minor issues. However it would be nice to have few more tests for bots mobiles especially popular ones like Googlebot. |
|
I like the introduction of kind. However except kind:bot it could be somewhat unreliable. Here are my concerns
What do you think? |
|
I'll have a look at the tests. I have a big json file with agent strings from the logs, I can extract some test data from that, or maybe the whole file could be used somehow? I agree with your concerns about mobile detection. I think that it could be improved somewhat, but it'll never be 100%, so it should serve as an indication only. If I merge the Android pull, detect Windows Phones and weed out iPads, I think that would be good enough for statistic purposes. For serving different content, there will always be edge cases. I looked at your second link, and it's one huge regex. Not the way to go. Another issue I'd like your opinion about. Right now some bots are split up for the image, mobile, etc. versions. Maybe it'd be better to merge these to a single bot? Not so sure about the google feed fetcher. It's from google, but actually a different thing. |
|
What's the deal with Safari on Linux, Android and other things that are not Mac or Windows? Shouldn't that be some other Webkit derivative? |
iPad is actually detected as a desktop because neither Safari of iPad are marked mobile, but iPhone is.
there is no real version number.
|
@pepijndevos thanks again for all the work. I would merge you patch however I will revert "kind" support. It is not reliable and would set incorrect expectations. For bots I will add a new attribute object.bot = T/F. I didn't understand the Webkit derivative question. |
|
Oh... I added some more tests. I think I'm done now. I understand your view on mobile detection... There is no Safari for Linux or Android, so these are other browsers based on Webkit. |
|
Actually Windows Phone is not yet added either. |
|
Are you planning to add Windows Phone? |
|
Working on it... should I make a new pull, or will you just pull it from my fork anyway? |
|
I can pull from you fork. |
|
Ok done. With this change, I'd say mobile detection is quite reliable. |
|
With Mobile detection issue is Tablets detected as mobile. I am not convinced with that. I have pushed my changes. I will pull your windows phone changes and then will release new version late today. |
|
Cool. Well, it properly detects iPads, Windows tablets, PlayBooks. Android is more tricky. In general the thing to look for is Mobile Safari, but this is not a 100% guarantee. But if we add Mobile Safari as a Mobile browser and make Android not mobile, that would catch all phones and very few Android tablets. |
I spent a good deal of time adding bots and phones until even myself could not tell what that agent string meant. What is left is a pile of unintelligible mess.
With that said, this patch supports every bot and feature phone that provided some useful info and occured more than a few times in a day worth of logs on a busy website.
One possible improvement would bo to assign types to browsers, so I can easily discern the bots from the phones from the desktops.