This is a free open source script licensed under the Creative Commons Attribution 4.0 International Licensender
- author : Morad Edwar
- license : Creative Commons
- version : 1.0
- email : me@morad-edwar.com
- status : Beta
1.0
You just need to modify :
start_url = 'http://www.domain.com'
domain = 'www.domain.com'
sitemap_path = '/tmp/sitemap.xml'
frequency = 'Daily'
priority = 'None'
ignore = ['.jpg','.png','/user?id=','login','logout']
Then run it
$ python sitemap.py
- White list
- Download delay