Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[软技能] 第89天 说说你对robots文件的理解，它有什么作用？ #746

Open
haizhilin2013 opened this issue Jul 13, 2019 · 1 comment
Open

[软技能] 第89天 说说你对robots文件的理解，它有什么作用？ #746

haizhilin2013 opened this issue Jul 13, 2019 · 1 comment
Labels
软技能 软技能

Comments

@haizhilin2013
Copy link
Collaborator

第89天 说说你对robots文件的理解，它有什么作用？
@haizhilin2013 haizhilin2013 added the 软技能 软技能 label Jul 13, 2019
@xxf1996
Copy link

xxf1996 commented Jul 14, 2019

robots.txt文件对抓取工具（如爬虫）在访问路径等方面做出了规定，用于阻止或引导抓取工具对网站下特定内容的抓取；

参考文档：Robots.txt 规范  |  搜索 |  Google Developers

@haizhilin2013 haizhilin2013 mentioned this issue Apr 20, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
软技能 软技能
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@haizhilin2013 @xxf1996