Python 3 robots.txt_scanner

###############################

crawler robots.txt

###############################

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
robots.py		robots.py