Skip to content

Conversation

summercms
Copy link
Contributor

@summercms summercms commented Oct 9, 2020

See github issue: #540

Some notes on this bot.

This bot says it's from aspiegel and from day one we have seen it coming from HUAWEI CLOUD servers! After the USA started banning Huawei they admitted that this bot is from them and it's going to be there search engine, see here: https://consumer.huawei.com/en/mobileservices/search/

Screenshot as they will probably take down their webpage:

image

Since the USA started trying to ban Huawei and TikTok etc. this bot has been hitting our test servers over 1000 times a day!

We label this bot as bad and their user agent keeps changing as time goes on. Sometimes they add fake user agents to hide, yet the same ip address then hits your server again in less than a second and then displays a user agent.

The regex code in this pr we have added code to allow a bot version number. In the future PetalBot may be like this: PetalBot/1.0 therefore we decided to add a version number now.

Note this bot is always changing and is crawling the internet to gather more information than it needs to do it's search engine job! Clearly it is data gathering for the Chinese government. The search engine aims to crawl apps only! Yet it keeps coming back to our test servers hitting them thousands of times a day!


User agents hitting our test servers:

UA:

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)

(compatible;PetalBot;+https://aspiegel.com/petalbot)

@coveralls
Copy link

coveralls commented Oct 9, 2020

Coverage Status

Coverage increased (+0.02%) to 99.97% when pulling fc122e9 on ayumi-cloud:PetalBot into 880b9fa on WhichBrowser:master.

NielsLeenheer added a commit that referenced this pull request Jan 4, 2021
@NielsLeenheer
Copy link
Member

Fixed in 6628eac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants