Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

添加robots爬虫文件 #42

Merged
merged 2 commits into from May 16, 2019
Merged

添加robots爬虫文件 #42

merged 2 commits into from May 16, 2019

Conversation

yJunS
Copy link
Member

@yJunS yJunS commented May 15, 2019

No description provided.

zacker330
zacker330 previously approved these changes May 15, 2019
View reviewed changes
Copy link
Contributor

@zacker330 zacker330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LinuxSuRen
Copy link
Member

页面上有这么一个元素，不知道是否有影响

<meta name="ROBOTS" content="NOINDEX, NOFOLLOW">

@yJunS
Copy link
Member Author

yJunS commented May 16, 2019

页面里有这个会影响到爬虫
NOINDEX命令：告诉搜索引擎不允许抓取这个页面
NOFOLLOW命令：告诉搜索引擎不允许从此页找到链接、拒绝其继续访问。

@LinuxSuRen
Copy link
Member

LinuxSuRen commented May 16, 2019 via email

@yJunS
Copy link
Member Author

yJunS commented May 16, 2019

不是，根目录的robots.txt是针对全站的。页面里的这个是单指此页面的，爬虫一般都会先检查根目录的robots.txt，才会抓取页面信息

@LinuxSuRen
Copy link
Member

了解了，我把 meta 也修改了。

donhui
donhui approved these changes May 16, 2019
View reviewed changes
Copy link
Member

@donhui donhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yJunS
Copy link
Member Author

yJunS commented May 16, 2019

LGTM

@LinuxSuRen LinuxSuRen merged commit bb006be into jenkins-zh:master May 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LinuxSuRen LinuxSuRen LinuxSuRen approved these changes

@donhui donhui donhui approved these changes

@zacker330 zacker330 zacker330 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@yJunS @LinuxSuRen @zacker330 @donhui