New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more search engine bots to robots list #22368

Merged
merged 8 commits into from Nov 1, 2018

Conversation

Projects
None yet
8 participants
@jurihahn
Contributor

jurihahn commented Sep 25, 2018

Pull Request for Issue # .

Summary of Changes

Add more robots to list for detection. List generated from:
https://github.com/monperrus/crawler-user-agents

Testing Instructions

Code review

Expected result

more robots in the robots list

Actual result

small robots list

Documentation Changes Required

No

@brianteeman

This comment has been minimized.

Contributor

brianteeman commented Sep 25, 2018

There must be a more up to date list available somewhere that can be used - for example I know that majestic-12 is one of the top (if not the top) crawlers

@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Sep 25, 2018

There must be a more up to date list available somewhere that can be used - for example I know that majestic-12 is one of the top (if not the top) crawlers

Should i completely update the list with one PR?

@brianteeman

This comment has been minimized.

Contributor

brianteeman commented Sep 25, 2018

If you can find a good list then it would make sense to do it all in one pr

@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Sep 25, 2018

@jurihahn jurihahn changed the title from Add BingBot to robots list to [WIP] Add more search engine bots to robots list Sep 25, 2018

@brianteeman

This comment has been minimized.

Contributor

brianteeman commented Sep 25, 2018

I note that that list doesn't include bingpreview which is in this pr

Found this list from keycdn of the most popular - lets just make sure we have those
https://www.keycdn.com/blog/web-crawlers/

Juri Hahn added some commits Sep 25, 2018

@SharkyKZ

This comment has been minimized.

Contributor

SharkyKZ commented Sep 25, 2018

This is a decent resource https://github.com/monperrus/crawler-user-agents. If not overkill.

Juri Hahn added some commits Sep 26, 2018

@jurihahn jurihahn changed the title from [WIP] Add more search engine bots to robots list to Add more search engine bots to robots list Sep 26, 2018

@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Sep 26, 2018

i think it's ready to test/review

Juri Hahn
@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Sep 26, 2018

This is a decent resource https://github.com/monperrus/crawler-user-agents. If not overkill.

i generated robot list from it

@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Sep 27, 2018

@SharkyKZ @brianteeman it's now OK?

@brianteeman

This comment has been minimized.

Contributor

brianteeman commented Sep 27, 2018

seems ok to me

@Quy

This comment has been minimized.

Contributor

Quy commented Sep 30, 2018

I have tested this item successfully on 12d17fa


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22368.

@sandewt

This comment has been minimized.

Contributor

sandewt commented on libraries/src/User/UserHelper.php in 12d17fa Sep 30, 2018

Line 760: Should be

->where($db->quoteName('user_id') . ' = ' . (int) $userId);
@jurihahn

This comment has been minimized.

Contributor

jurihahn commented Oct 2, 2018

@sandewt wrong thread?

@sandewt

This comment has been minimized.

Contributor

sandewt commented Oct 4, 2018

libraries/src/User/UserHelper.php, Line 760: Should be ->where($db->quoteName('user_id') . ' = ' . (int) $userId);

@jurihahn see, [#22079] - typecasting code privacy content plugin


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22368.

@infograf768

This comment has been minimized.

Member

infograf768 commented Oct 4, 2018

@sandewt
This PR does not touch at this file.
Please create a new issue or PR for that one.

@sandewt

This comment has been minimized.

Contributor

sandewt commented Oct 4, 2018

This PR does not touch at this file.

Thanks @infograf768 and @jurihahn.

I got it.

There are more files in the Joomla core with this issue.

So it should be a new PR.

[EDIT] Some text.

@brianteeman

This comment has been minimized.

Contributor

brianteeman commented Oct 17, 2018

I have tested this item successfully on 12d17fa


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22368.

@Quy

This comment has been minimized.

Contributor

Quy commented Oct 17, 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22368.

@joomla-cms-bot joomla-cms-bot added the RTC label Oct 17, 2018

@mbabker mbabker added this to the Joomla 3.9.1 milestone Nov 1, 2018

@mbabker mbabker merged commit ebdd957 into joomla:staging Nov 1, 2018

4 of 5 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
Hound No violations found. Woof!
JTracker/HumanTestResults Human Test Results: 2 Successful 0 Failed.
Details
continuous-integration/drone/pr the build was successful
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joomla-cms-bot joomla-cms-bot removed the RTC label Nov 1, 2018

@jurihahn jurihahn deleted the jurihahn:add-bingbot-to-robots-list branch Nov 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment